A new model is cheaper and answers differently
Changing the large language model (LLM) behind a feature means running the same cases on both versions and moving users over slowly. It matters because a new version can be better on average and still worse on some cases, and your users find those first.
No lab here trains or serves a model, so this lesson compares two recorded versions. The method is the one you would use for real.
By the end of this lesson
Note
I can decide from two recorded runs whether to move a feature to another model.
Key words
| Word | What it means |
|---|---|
| Regression | a case that passed before and fails now |
| Canary | a small share of users on the new version first |
| Rollback | the way back to the old version |
| Bucket | a fixed number a user always falls into |
| Cost rise | the percentage more the new version costs |
The cells students fill in are left blank in this preview.
Run both versions on the same cases
Use the same cases for both versions, and record for each case whether it passed and what it cost. Compare the pass counts, but also list the regressions.
Two versions with the same pass count can differ: one may fail a case the other passed. A regression on an important case can matter more than a better average.
Worked example: old and new on six cases
| Version | Passed | Cost | Regressions |
|---|---|---|---|
| Old | 5 of 6 | 180 | none |
| New | 5 of 6 | 144 | c3 passed before and fails now |
The cells students fill in are left blank in this preview.
Move a small share first
Do not switch everyone at once. Send a small share of users to the new version and watch. Choose users by a fixed rule, so the same person always gets the same version.
Keep the old version running so you can go back in one step. And look at the bill: a price change can wipe out the gain, so state the cost rise you will accept before you look.
A common mistake
Warning
Many people think that a newer model is always better, so a quick test is enough. A new model can be better on average and worse on your cases. Run your own cases, move a small share first, and keep the way back.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' '' '// Two recorded versions of one feature on the same six cases. Cost is micro-euros.' 'const oldRun = [1, 1, 1, 0, 1, 1].map(function (ok, i) { return { case: \"c\" + (i + 1), ok: ok === 1, cost: 30 }; });' 'const newRun = [1, 1, 0, 1, 1, 1].map(function (ok, i) { return { case: \"c\" + (i + 1), ok: ok === 1, cost: 24 }; });' 'const MAX_REGRESSIONS = 1;' 'function passes(run) { return run.filter(function (r) { return r.ok; }).length; }' 'function total(run) { return run.reduce(function (sum, r) { return sum + r.cost; }, 0); }' > change.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function regressions(a, b) {' ' return a.filter(function (r, i) { return r.ok && b[i].ok === false; }).map(function (r) { return r.case; });' '}' '// A user is in the canary share when the letters of the name add up to a bucket below the percent.' 'function inCanary(user, percent) {' ' let n = 0;' ' for (const ch of user) n = n + ch.charCodeAt(0);' ' return n % 100 < percent;' '}' >> change.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"old: \" + passes(oldRun) + \" of 6, cost \" + total(oldRun));' ' console.log(\"new: \" + passes(newRun) + \" of 6, cost \" + total(newRun));' ' const bad = regressions(oldRun, newRun);' ' console.log(\"regressions: \" + bad.join(\", \") + (bad.length <= MAX_REGRESSIONS ? \" (allowed)\" : \" (too many)\"));' ' const users = [\"ana\", \"cy\", \"eva\", \"gus\", \"li\", \"zoe\"];' ' console.log(\"canary 10 percent: \" + users.filter(function (u) { return inCanary(u, 10); }).join(\", \"));' ' const r = await ask([{ role: \"user\", content: \"What is 15 percent of 200 euros? Answer in one short sentence.\" }], \"15 percent of 200 euros is 30 euros.\", 40);' ' console.log(\"Today live (not in the recording): \" + r.text);' '}' 'main();' >> change.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the comparison
docker exec -w /home/node/project agent node change.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it shows |
|---|---|
| old: 5 of 6, cost 180 | The recorded old version |
| new: 5 of 6, cost 144 | The recorded new version, cheaper |
| regressions: c3 (allowed) | One regression, within the limit of one |
| canary 10 percent: ana | The user the fixed rule puts in the small share |
| Today live (not in the recording): ... | Your own call. It differs each run, or says Recorded sample |
The cells students fill in are left blank in this preview.