How to review a change plan
A change plan is a short document saying what will change, why it is safe and how to undo it. A change review checks the plan before anything ships.
It matters because a bad AI change can reach every user in minutes. A checklist written in advance gives a clear answer: go, needs more, or stop.
By the end of this lesson
Note
I can review a change plan before it ships.
Key words
| Word | What it means |
|---|---|
| Change plan | a short document that says what changes and how to undo it |
| Evidence | results that show the change is no worse |
| Rollout | how the change reaches users, in stages |
| Way back | the steps that undo the change |
| Approver | the person who says yes |
The cells students fill in are left blank in this preview.
A plan needs five parts
The five parts are the change, the evidence, the rollout, the way back and the approver. A plan with a missing part is not ready, and the reviewer says which part.
Check for missing parts first. There is no point checking the numbers of a part that is not there.
Each part has a written limit
Write the limits before you read any plan. The evidence needs at least 100 cases and no drop in the pass rate. The rollout starts at 10 percent or less, with a minimum of 50 requests.
The way back is tested and takes 15 minutes or less. The approver is not the author.
Worked example: two plans against the limits
| Rule | Prompt v7 | Model swap |
|---|---|---|
| At least 100 cases | 240, passes | 60, fails |
| No drop in pass rate | 71 to 74, passes | 80 to 76, fails |
| Start at 10 percent or less | 5, passes | 50, fails |
| Minimum of 50 requests | 50, passes | 0, fails |
| Way back tested, 15 minutes or less | Yes, 10 | No, 30. Fails twice |
| Approver is not the author | Sam, passes | Ria wrote it, fails |
| Decision | Go | Stop |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a plan with good numbers is ready. Numbers do not show that the way back works. A way back that was never tried may fail when it is needed. So the review asks whether it was tested, and how long it took.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the review script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const plans = {' ' \"prompt v7\": { change: \"new refund prompt\", evidence: { cases: 240, before: 71, after: 74 },' ' rollout: { start: 5, min: 50 }, back: { minutes: 10, tested: true }, approver: \"Sam\", author: \"Ria\" },' ' \"model swap\": { change: \"new model\", evidence: { cases: 60, before: 80, after: 76 },' ' rollout: { start: 50, min: 0 }, back: { minutes: 30, tested: false }, approver: \"Ria\", author: \"Ria\" },' ' \"tone tweak\": { change: \"friendlier tone\", evidence: { cases: 150, before: 70, after: 70 },' ' rollout: { start: 10, min: 50 }, approver: \"Sam\", author: \"Ria\" },' '};' > review.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the review script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function problems(p) {' ' const out = [\"change\", \"evidence\", \"rollout\", \"back\", \"approver\"].filter(function (k) { return p[k] === undefined; })' ' .map(function (k) { return k + \" missing\"; });' ' if (p.evidence && p.evidence.cases < 100) out.push(\"evidence too small\");' ' if (p.evidence && p.evidence.after < p.evidence.before) out.push(\"evidence shows a regression\");' ' if (p.rollout && p.rollout.start > 10) out.push(\"rollout starts too big\");' ' if (p.rollout && p.rollout.min < 50) out.push(\"rollout has no minimum\");' ' if (p.back && p.back.tested === false) out.push(\"back untested\");' ' if (p.back && p.back.minutes > 15) out.push(\"back too slow\");' ' if (p.approver === p.author) out.push(\"approver wrote it\");' ' return out;' '}' >> review.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the review script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const STOP = [\"evidence shows a regression\", \"back missing\", \"back untested\", \"back too slow\", \"approver wrote it\"];' 'function decide(p) {' ' const found = problems(p);' ' if (found.length === 0) return \"go\";' ' return found.some(function (f) { return STOP.indexOf(f) >= 0; }) ? \"stop\" : \"needs more\";' '}' 'async function main() {' ' Object.keys(plans).forEach(function (name) {' ' console.log(name + \": \" + decide(plans[name]) + \" \" + JSON.stringify(problems(plans[name])));' ' });' ' const q = \"In one short sentence, why must the person who approves a change not be its author?\";' ' const sample = \"A second pair of eyes catches what the author cannot see in their own work.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> review.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the change review
docker exec -w /home/node/project agent node review.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| prompt v7: go [] | No problems at all |
| model swap: stop [...] | Seven problems, and some are safety rules |
| tone tweak: stop ["back missing"] | No way back at all |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.