Did the new prompt make it worse?
A regression run compares two versions of a feature on the same cases and says what got worse. It matters because a prompt edit that fixes one message often breaks another, and nobody sees it.
The shop has a new version of its sorter. The pass rate looks the same as before. That does not mean nothing changed.
By the end of this lesson
Note
I can compare two versions on the same cases and decide whether one is worse.
Key words
| Word | What it means |
|---|---|
| Regression | a case that passed before and fails now |
| Pass rate | the share of cases that pass |
| Baseline | the version you compare against |
| Noise | change between runs of one unchanged version |
| Unstable case | a case that passes in some runs and fails in others |
The cells students fill in are left blank in this preview.
Compare the same cases, version by version
Run both versions over the same golden set and keep the result of each case. The pass rate gives one number. The list of cases that changed gives the reason.
A regression is a case that passed in the baseline and fails now. Read that list before you read the rate.
Worked example: the same rate, different cases
| Version | Cases passed of 12 | Pass rate | Cases that fail |
|---|---|---|---|
| Old, run 1 | 10 | 83 percent | 8, 12 |
| Old, run 2 | 9 | 75 percent | 5, 11, 12 |
| New | 9 | 75 percent | 4, 10, 12 |
The cells students fill in are left blank in this preview.
One run proves little, because replies vary
A model may answer one message differently each time. So run the old version twice. The gap between those two pass rates is the noise.
An unstable case passes in one run and fails in the next. Do not blame the new version for it.
Decide with the rate, then with the cases
If the new rate is lower than the old one by more than the noise, call it worse. If it is inside the noise, the rate says nothing.
Then look at the cases. A case that passed in every old run and fails in the new one is a firm regression, and it is the one to fix.
Write the three recorded runs, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const want = [\"billing\", \"delivery\", \"other\", \"billing\", \"delivery\", \"other\", \"billing\", \"delivery\", \"unknown\", \"unknown\", \"billing\", \"other\"];' '// Recorded runs: a label and an urgency for each case, and a text where it matters.' 'const runA1 = [[\"billing\", 2], [\"delivery\", 2], [\"other\", 1], [\"billing\", 2], [\"delivery\", 3], [\"other\", 1], [\"billing\", 2], [\"billing\", 2], [\"unknown\", 1], [\"unknown\", 1], [\"billing\", 3], [\"delivery\", 2]];' > runs.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the three recorded runs, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const runA2 = [[\"billing\", 2], [\"delivery\", 2], [\"other\", 1], [\"billing\", 2], [\"other\", 1], [\"other\", 1], [\"billing\", 2], [\"delivery\", 2], [\"unknown\", 1], [\"unknown\", 1], [\"other\", 2], [\"delivery\", 2]];' >> runs.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the three recorded runs, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const runB = [[\"billing\", 2], [\"delivery\", 2], [\"other\", 1], [\"other\", 2], [\"delivery\", 3], [\"other\", 1], [\"billing\", 2], [\"delivery\", 2], [\"unknown\", 1], [\"unknown\", 1, \"Your refund has been approved.\"], [\"billing\", 3], [\"delivery\", 2]];' 'module.exports = { want, runA1, runA2, runB };' >> runs.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { want, runA1, runA2, runB } = require(\"./runs\");' '// The rules of lesson 2, shortened: right label, urgency 1 to 3, no forbidden phrase.' 'function passes(entry, expected) {' ' const said = (entry[2] || \"\").toLowerCase();' ' return entry[0] === expected && entry[1] >= 1 && entry[1] <= 3 && said.includes(\"refund has been\") === false;' '}' 'function results(run) {' ' return run.map(function (entry, at) { return passes(entry, want[at]); });' '}' > compare3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function rate(list) {' ' return Math.floor(list.filter(Boolean).length * 100 / list.length);' '}' 'const a1 = results(runA1);' 'const a2 = results(runA2);' 'const b = results(runB);' 'console.log(\"Pass rate A1: \" + rate(a1) + \" percent\");' 'console.log(\"Pass rate A2: \" + rate(a2) + \" percent\");' 'console.log(\"Pass rate B: \" + rate(b) + \" percent\");' 'const noise = Math.abs(rate(a1) - rate(a2));' 'console.log(\"Noise between A1 and A2: \" + noise + \" points\");' >> compare3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the comparison, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const low = Math.min(rate(a1), rate(a2));' 'const high = Math.max(rate(a1), rate(a2));' 'let verdict = \"no clear change\";' 'if (rate(b) < low - noise) verdict = \"worse\";' 'if (rate(b) > high + noise) verdict = \"better\";' 'console.log(\"Verdict from the rates alone: \" + verdict);' 'const worse = [];' 'const unstable = [];' 'const always = [];' 'for (let at = 0; at < want.length; at = at + 1) {' ' const same = a1[at] === a2[at];' ' if (same === false) unstable.push(at + 1);' ' else if (a1[at] && b[at] === false) worse.push(at + 1);' ' else if (a1[at] === false && b[at] === false) always.push(at + 1);' '}' 'console.log(\"Regressions (pass in both A runs, fail in B): \" + worse.join(\", \"));' 'console.log(\"Unstable (differ between A1 and A2): \" + unstable.join(\", \"));' 'console.log(\"Fail in every run: \" + always.join(\", \"));' >> compare3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Compare the versions
docker exec -w /home/node/project agent node compare3.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the comparison prints
| Line | What it shows |
|---|---|
| Pass rates | A1 83, A2 75, B 75 percent |
| Noise | 8 points |
| Verdict from the rates | no clear change |
| Regressions | cases 4 and 10 |
| Unstable | cases 5, 8 and 11 |
| Fail in every run | case 12 |
The cells students fill in are left blank in this preview.
Try it
Tip
In runs.js change the urgency of case 1 in runB from 2 to 9, then compare again. Case 1 should join the regressions.
A common mistake
Warning
Many people compare one run of each version and trust the difference. With replies that vary, one run can mislead in either direction. Run the baseline at least twice, measure the noise and judge a change only against it.