How a circuit breaker works
A circuit breaker is a small rule that stops calling a service that keeps failing. While it is open, the feature uses a fallback instead of waiting for a call that will fail.
The module on cost, tracing and reliability lists the fallbacks. This lesson goes further: when to stop calling, when to try again, and a kill switch that a person can press.
By the end of this lesson
Note
I can keep a feature useful while its model is failing.
Key words
| Word | What it means | Example |
|---|---|---|
| Circuit breaker | a rule that stops calls to a failing service | |
| Closed | the normal state: calls go through | |
| Open | the stopped state: no calls, a fallback answers | |
| Half open | one trial call to see if the service is back | |
| Cool down | steps to wait before a trial | 4 |
| Kill switch | a control that turns a feature off at once |
The cells students fill in are left blank in this preview.
Three states
A breaker is closed when all is well. Calls go through. It counts failures in a row. When the count reaches the threshold, here 3, it opens.
While open it makes no call. After the cool down, here 4 steps, it goes half open and lets one call through. A success closes it. A failure opens it again.
Worked example: steps 3 to 9
| Step | Call would | State found | Served | Why |
|---|---|---|---|---|
| 3 | Fail | Closed | Failed | First failure in a row, count 1 |
| 4 | Fail | Closed | Failed | Count 2 |
| 5 | Fail | Closed | Failed | Count reaches 3, so it opens |
| 6 | Work | Open | Skipped | No call. 1 step since it opened |
| 7 | Work | Open | Skipped | 2 steps since it opened |
| 8 | Work | Open | Skipped | 3 steps since it opened |
| 9 | Work | Half open | Model | 4 steps, the trial works, so it closes |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a breaker should open on the first failure. One failure is often a blip. Count failures in a row, and let a success reset the count, or a healthy service gets shut off by a single bad call.
What the fallback says
A fallback should be honest and useful. The best one is a saved answer to the same question. If there is none, say plainly that the assistant is busy and when to try again.
Never show a made-up answer as if the model gave it.
A kill switch is a person's decision
A breaker reacts to failures. A kill switch is pressed by a person when the model is working but should not be used. It might give wrong answers, or cost too much.
While it is on, every call is skipped. It does not touch the breaker, so the breaker keeps its own count and state.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the breaker script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'function step(b, outcome, i, threshold, cooldown, kill) {' ' if (kill) return [b.state, \"skipped\"];' ' if (b.state === \"open\" && i - b.openedAt >= cooldown) b.state = \"half_open\";' ' const found = b.state;' ' if (found === \"open\") return [found, \"skipped\"];' ' if (outcome === 1) { b.state = \"closed\"; b.fails = 0; return [found, \"model\"]; }' ' b.fails = b.fails + 1;' ' if (found === \"half_open\" || b.fails >= threshold) { b.state = \"open\"; b.openedAt = i; b.fails = 0; }' ' return [found, \"failed\"];' '}' > breaker.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the breaker script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const outcomes = [1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1];' 'function drive(kills) {' ' const b = { state: \"closed\", fails: 0, openedAt: 0 };' ' return outcomes.map(function (o, k) {' ' const r = step(b, o, k + 1, 3, 4, kills.includes(k + 1));' ' return r[0] + \":\" + r[1];' ' });' '}' >> breaker.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the breaker script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const trace = drive([]);' ' console.log(trace.slice(0, 10).join(\" \"));' ' console.log(trace.slice(10).join(\" \"));' ' const made = trace.filter(function (t) { return t.endsWith(\"skipped\") === false; }).length;' ' console.log(\"calls made \" + made + \" of \" + trace.length);' ' console.log(\"kill 9 and 10: \" + drive([9, 10]).slice(8, 11).join(\" \"));' ' const q = \"In one short sentence, what should a feature say when its model is down?\";' ' const reply = await ask([{ role: \"user\", content: q }], \"Say plainly that the assistant is busy, and offer a saved answer or another way.\", 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> breaker.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the breaker
docker exec -w /home/node/project agent node breaker.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| closed:model closed:model closed:failed ... | Steps 1 to 10, the state found and how it was served |
| open:skipped open:skipped open:skipped half_open:model | The breaker opens, waits, and a trial works |
| calls made 14 of 20 | Six calls were saved |
| kill 9 and 10: open:skipped open:skipped half_open:model | The switch skipped two steps and the cool down carried on |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.