Keep working when a model fails
A fallback is what a feature does when a model call fails, so the user still gets something. It matters because model calls fail in several different ways, and one answer to all of them is wrong for most.
A timeout deserves a retry. A refusal does not. An empty daily budget needs a cached answer or an honest message. Choose per failure.
By the end of this lesson
Note
I can keep a feature working when a model call fails by choosing a fallback for each kind of failure.
Key words
| Word | What it means |
|---|---|
| Fallback | what you do instead when a call fails |
| Retry | asking the same model again, after a wait |
| Second model | a different model asked when the first fails |
| Plain answer | a fixed, honest message with no model |
| Degrade | to give a smaller answer rather than none |
| Failure kind | the reason a call failed |
The cells students fill in are left blank in this preview.
A fallback is a chain, in order
Put the options in order, from the best answer to the most degraded. First a retry, after a wait. Then a second model. Then a cached answer, which may be old. Last a plain answer: a fixed message that is true and tells the user what to do.
Each step has a cost in time and money. A chain that tries everything on every failure doubles your bill during an outage.
Worked example: which fallback fits
| Failure | First | Why |
|---|---|---|
| Timeout, rate limit | Retry once | Often passes by itself |
| Server error | Second model | The first one is down |
| Bad format | Retry once | A second try often parses |
| Refusal | Plain answer | Asking again will not change it |
| Budget used up | Cached answer | No call is allowed at all |
The cells students fill in are left blank in this preview.
Bound the chain and say so
Put a limit on every step: one retry, one second model. Wait between tries, and count the wait against the user's timeout. Never let a fallback hide a failure. Log which step answered.
A cached answer may be out of date, so say so if it matters. A plain answer must never pretend to come from the model.
A common mistake
Warning
Many people think retrying is always the safest fallback. But retrying a refusal wastes money, and retrying during an outage adds load. Retry only failures that can pass by themselves, and only once.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the fallback chain, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '' 'let now = 0; // milliseconds, on a clock we move ourselves' 'const store = new Map([[\"opening hours\", \"We open at nine and close at five.\"]]);' 'const PLAIN = \"Sorry, the assistant cannot answer now. Please try again later.\";' '// A recorded failure stands in for a real one. Nothing here causes a real error.' 'function fail(kind) { throw new Error(kind); }' > fallback5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the fallback chain, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function secondModel() {' ' const messages = [{ role: \"user\", content: \"State opening hours: nine to five, in one sentence.\" }];' ' return (await ask(messages, \"We open at nine and close at five.\", 40)).text;' '}' >> fallback5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the fallback chain, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function answer(question, second) {' ' const steps = [' ' [\"first model\", function () { fail(\"rate_limit\"); }],' ' [\"retry\", function () { now = now + 200; fail(\"rate_limit\"); }],' ' [\"second model\", second],' ' [\"cache\", function () { return store.has(question) ? store.get(question) : fail(\"miss\"); }],' ' ];' >> fallback5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the fallback chain, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' for (const [name, run] of steps) {' ' try {' ' const text = await run();' ' console.log(question + \": answered by \" + name + \" at \" + now + \" ms\");' ' return text;' ' } catch (error) {' ' console.log(question + \": \" + name + \" failed (\" + error.message + \")\");' ' }' ' }' ' console.log(question + \": plain answer\");' ' return PLAIN;' '}' >> fallback5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the fallback chain, part 5
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const down = function () { fail(\"server_error\"); };' ' await answer(\"opening hours\", secondModel);' ' await answer(\"opening hours\", down);' ' await answer(\"refund policy\", down);' '}' 'main();' >> fallback5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the chain
docker exec -w /home/node/project agent node fallback5.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
The failures are recorded
Nothing in this Run causes a real failure. The rate limits and the server error are written into the script as recorded data, and the clock is a number it moves. Only the second model is a real call.
A real system gets these errors from a real network, and sleeps for real between tries.
One possible output
| Line | What it shows |
|---|---|
| opening hours: first model failed (rate_limit) | A recorded failure |
| opening hours: retry failed (rate_limit) | The retry, after 200 ms on our clock |
| opening hours: answered by second model at 200 ms | The one real call |
| opening hours: answered by cache at 400 ms | The second model is down, so the cache answers |
| refund policy: plain answer | Nothing was cached, so the plain message is used |
The cells students fill in are left blank in this preview.