What to do when an AI feature fails
An incident is a time when a feature harms users or the business, such as wrong refunds or a leaked address. Running an incident means finding it, limiting it, fixing it and learning from it.
It matters because a calm, ordered response limits the harm. A good review then stops the same failure happening again.
By the end of this lesson
Note
I can run an AI incident and learn from it.
Key words
| Word | What it means | Example |
|---|---|---|
| Incident | a time a feature is harming users | |
| Timeline | the events in order, with their times | |
| Severity | how serious it is, from 1 to 3 | 2 |
| Mitigate | to limit the harm before the cause is fixed | |
| Review | a meeting to learn what let it happen |
The cells students fill in are left blank in this preview.
Start with a timeline from the records
Put every event in order of time, even if the records arrive in a different order. Each line says when, who and what.
Then work out two numbers. Time to detect is the gap from the start to the first sign. Time to recover is the gap from the start to the fix.
Worked example: a wrong refund prompt
| Minute | Who | Event | What it gives |
|---|---|---|---|
| 0 | release | New prompt goes live | The start |
| 25 | support | First customer report | Time to detect is 25 |
| 31 | on-call | Incident declared | Work begins |
| 40 | on-call | Old prompt restored | Harm stops |
| 52 | monitor | Refunds back to zero | Time to recover is 52 |
The cells students fill in are left blank in this preview.
Severity sets who does what
Write the severity rule before an incident. Severity 1 is private data shown, or half the users or more. Severity 2 is ten percent or more. Anything less is 3.
Each severity has a short list of actions, and each action has a role: lead, fixer or comms. Roles keep people from all doing the same job.
Worked example: severity and actions
| Severity | When | Actions |
|---|---|---|
| 1 | Private data, or 50 percent or more | Page the owner, roll back, tell users, hold a review |
| 2 | 10 to 49 percent | Roll back, tell the team, hold a review |
| 3 | Under 10 percent | Note it in the log |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a review should find who made the mistake. That makes people hide things next time. Ask what in the system let it through: no canary, no alert, a rule nobody checked. Write roles in the review, and never blame a person.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the incident script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const events = [' ' { at: 31, who: \"on-call\", kind: \"declared\", what: \"incident declared\" },' ' { at: 0, who: \"release\", kind: \"start\", what: \"new prompt goes live\" },' ' { at: 52, who: \"monitor\", kind: \"resolved\", what: \"wrong refunds back to zero\" },' ' { at: 25, who: \"support\", kind: \"detected\", what: \"first customer report\" },' ' { at: 40, who: \"on-call\", kind: \"mitigated\", what: \"old prompt restored\" },' '];' > incident.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the incident script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function first(kind) { return events.filter(function (e) { return e.kind === kind; })[0].at; }' 'function severity(percent, privateData) {' ' if (privateData || percent >= 50) return 1;' ' return percent >= 10 ? 2 : 3;' '}' 'const sorted = events.slice().sort(function (a, b) { return a.at - b.at; });' >> incident.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the incident script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' sorted.forEach(function (e) { console.log(\"minute \" + e.at + \" \" + e.who + \": \" + e.what); });' ' console.log(\"time to detect: \" + (first(\"detected\") - first(\"start\")) + \" minutes\");' ' console.log(\"time to recover: \" + (first(\"resolved\") - first(\"start\")) + \" minutes\");' ' console.log(\"severity: \" + severity(18, false));' ' const q = \"In one short sentence, why should an incident review name roles and not people?\";' ' const sample = \"People hide mistakes when they are blamed, and the review must find what the system allowed.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> incident.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the incident report
docker exec -w /home/node/project agent node incident.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| minute 0 release: new prompt goes live | Five events, sorted by minute |
| time to detect: 25 minutes | First report minus the start |
| time to recover: 52 minutes | Resolved minus the start |
| severity: 2 | From 18 percent affected, no private data |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.