Failing a build on a bad prompt
A team changes a prompt on Friday. On Monday, customers see worse answers. Nobody looked at the score, because nothing made them look.
A regression test in a build stops that. The module on evals and testing compares two runs by hand. This lesson makes the comparison stop a change automatically.
By the end of this lesson
Note
I can make a worse prompt fail a build.
Key words
| Word | What it means | Example |
|---|---|---|
| Build | the automatic checks that run on a change | |
| Frozen set | an evaluation set that does not change between runs | |
| Pass mark | the lowest pass rate a change may have | 80 |
| Critical case | a case that may never go from pass to fail | |
| Exit code | a number a program ends with; zero means fine |
The cells students fill in are left blank in this preview.
A frozen set and a pass mark
The build runs your feature over the same frozen set every time. If the set changed with the prompt, you could not compare two runs.
You then pick a pass mark, such as 80 percent. A change that scores under the mark fails the build.
The cases that changed matter most
A rate hides movement. One case can break while another is fixed, and the rate stays the same.
So the build also lists the cases that changed. A broken case passed before and fails now. A fixed case does the opposite.
Worked example: twelve cases, two runs
| Measure | Old run | New run |
|---|---|---|
| Cases passed | 10 of 12 | 9 of 12 |
| Pass rate, whole percent | 83 | 75 |
| Broken cases | - | 4 and 7 |
| Fixed cases | - | 6 |
| Critical case broken | - | Yes, case 4 |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a pass rate above the mark means the change is safe. A critical case can break while the rate stays high. Check critical cases first and the rate second.
An exit code steers the build
A program ends with a number. Zero means fine and any other number means a problem. The build stops when it sees a number that is not zero.
We use 2 for a broken critical case, 1 for a rate under the mark and 0 for a pass. The critical check comes first.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the build gate, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const mark = 80;' 'const rows = [' ' \"1 c 1 1\", \"2 n 1 1\", \"3 n 1 1\", \"4 c 1 0\",' ' \"5 n 1 1\", \"6 n 0 1\", \"7 n 1 0\", \"8 n 1 1\",' ' \"9 c 1 1\", \"10 n 0 0\", \"11 n 1 1\", \"12 n 1 1\",' '].map(function (r) {' ' const p = r.split(\" \");' ' return { id: Number(p[0]), critical: p[1] === \"c\", old: p[2] === \"1\", now: p[3] === \"1\" };' '});' > gate.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the build gate, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function rate(key) {' ' const n = rows.filter(function (r) { return r[key]; }).length;' ' return Math.floor(n * 100 / rows.length);' '}' 'const broken = rows.filter(function (r) { return r.old && r.now === false; });' 'const fixed = rows.filter(function (r) { return r.old === false && r.now; });' 'function ids(list) {' ' return list.length === 0 ? \"none\" : list.map(function (r) { return r.id; }).join(\", \");' '}' >> gate.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the build gate, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function exitCode() {' ' if (broken.some(function (r) { return r.critical; })) return 2;' ' if (rate(\"now\") < mark) return 1;' ' return 0;' '}' 'async function main() {' ' console.log(\"pass rate: \" + rate(\"old\") + \" -> \" + rate(\"now\") + \" (mark \" + mark + \")\");' ' console.log(\"broken: \" + ids(broken) + \" fixed: \" + ids(fixed));' ' console.log(\"exit code: \" + exitCode());' ' const q = \"Write one calm sentence for a pull request: case 4 passed before and fails now.\";' ' const sample = \"Case 4 passed on the old prompt and fails on this one, so the build is stopped.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"comment: \" + reply.text);' '}' 'main();' >> gate.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the build gate
docker exec -w /home/node/project agent node gate.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the gate prints
| Line | What it shows |
|---|---|
| pass rate: 83 -> 75 (mark 80) | Old and new rate, and the mark |
| broken: 4, 7 fixed: 6 | The cases that changed |
| exit code: 2 | Case 4 is critical, so the build stops |
| comment: ... | A sentence from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.