How a canary rollout works
A canary rollout sends a small share of users to a change before everyone gets it. If the change is bad, only a few people meet the fault.
It works when the same users are in the canary every time. It also needs a rule for the decision, written before the change goes out.
By the end of this lesson
Note
I can send a small share of traffic to a change and decide by a rule.
Key words
| Word | What it means | Example |
|---|---|---|
| Canary | the small group that gets a change first | |
| Share | the percent of users in the canary | 5 |
| Bucket | a fixed number from 0 to 99 for each user | |
| Promote | to give the change to more users | |
| Roll back | to take the change away again | |
| Margin | how much worse the canary may be |
The cells students fill in are left blank in this preview.
A fixed bucket decides who is in
Do not pick users at random each time, or one person flips between the old and new version. Give each user a bucket from 0 to 99 that never changes.
A user is in the canary when their bucket is below the share. At a share of 20, buckets 0 to 19 are in.
Worked example: five users
| User | Bucket | In at share 5 | In at share 20 | In at share 50 |
|---|---|---|---|---|
| u7 | 4 | Yes | Yes | Yes |
| u9 | 18 | No | Yes | Yes |
| u15 | 33 | No | No | Yes |
| u1 | 62 | No | No | No |
| u10 | 98 | No | No | No |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a small canary with no errors proves the change is safe. Five users sending a few requests prove very little. Set a minimum number of requests first, and hold until the canary has that many.
Write the rule before the rollout
Compare the canary's error percent with the old version's on the same stage. Use whole-number division, rounded down.
Hold if the canary has fewer than 50 requests. Roll back if its percent is more than the old percent plus a margin of 2. Otherwise promote.
Worked example: three stages
| Share | Canary requests | Canary errors | Old percent | Decision |
|---|---|---|---|---|
| 5 | 40 | 1 | 1 | Hold, only 40 requests |
| 20 | 180 | 4 | 2 | Promote, 2 is not above 4 |
| 50 | 300 | 18 | 2 | Roll back, 6 is above 4 |
The cells students fill in are left blank in this preview.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the canary script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'function bucket(user) {' ' let s = 0;' ' for (let i = 0; i < user.length; i++) s = s + user.charCodeAt(i);' ' return (s * 7) % 100;' '}' 'function inCanary(user, share) { return bucket(user) < share; }' 'const users = [];' 'for (let i = 1; i <= 20; i++) users.push(\"u\" + i);' > canary.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the canary script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const stages = [' ' { share: 5, cn: 40, ce: 1, bn: 760, be: 15 },' ' { share: 20, cn: 180, ce: 4, bn: 640, be: 13 },' ' { share: 50, cn: 300, ce: 18, bn: 450, be: 9 },' '];' 'function pct(e, n) { return Math.floor(e * 100 / n); }' 'function decide(s) {' ' if (s.cn < 50) return \"hold\";' ' return pct(s.ce, s.cn) > pct(s.be, s.bn) + 2 ? \"rollback\" : \"promote\";' '}' >> canary.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the canary script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' [5, 20].forEach(function (share) {' ' console.log(\"share \" + share + \": \" + users.filter(function (u) { return inCanary(u, share); }).join(\" \"));' ' });' ' for (const s of stages) {' ' const d = decide(s);' ' console.log(\"stage \" + s.share + \": canary \" + pct(s.ce, s.cn) + \" percent, old \" + pct(s.be, s.bn) + \" percent -> \" + d);' ' if (d === \"rollback\" || d === \"hold\") break;' ' }' ' const q = \"In one short sentence, why send only a few users to a change first?\";' ' const sample = \"If the change is bad, only a few people meet the fault and you can undo it fast.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> canary.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the canary
docker exec -w /home/node/project agent node canary.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| share 5: u7 | The one user in the canary at 5 |
| share 20: u7 u8 u9 u11 u12 u13 u20 | Seven users. u7 is still in |
| stage 5: canary 2 percent, old 1 percent -> hold | Under 50 requests, so wait |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.