Reading eval results to choose a fix
Reading eval results means grouping the failures by kind, then changing the cheapest thing that fixes the most. It matters because every change helps some cases and can hurt others.
Fernside Library's first version passed 5 of 10 cases. The team has four ideas. Only one of them is worth shipping first, and the scores say which.
By the end of this lesson
Note
I can read eval results and decide what to change first.
Key words
| Word | What it means | Example |
|---|---|---|
| Failure kind | the reason a case failed | invented |
| Candidate | a change you are thinking of making | |
| Regression | a case that passed before and fails now | |
| Cost per 1000 | what 1000 answers cost to produce | |
| Cheapest fix | the change that costs least for the gain |
The cells students fill in are left blank in this preview.
Group the failures by kind
A score of 5 of 10 says how much is wrong, not why. Group the five failures by kind and the work becomes clear.
Each kind has its own cause and its own cheap fix. One fix for a whole kind is better than five fixes for five cases.
Worked example: the five failures
| Kind | Cases | Likely cause |
|---|---|---|
| Invented | c3, c7 | The prompt does not forbid guessing |
| Vague | c5 | The question was answered in general words |
| Refusal | c8 | The prompt makes it too careful |
| Long | c9 | Nothing limits the length |
The cells students fill in are left blank in this preview.
Try the cheapest fix first
Rank the candidates by what they cost. A new line in the prompt costs a little on every answer, because the prompt grows. A bigger model costs about three times as much on every answer.
Run the cheapest first and score it. Stack the next one on top only if the score asks for it.
Worked example: four candidates
| Run | Change | Passed | Cost per 1000 |
|---|---|---|---|
| v1 | The first version | 5 | 40 |
| v2 | Use only figures from the sheet | 7 | 42 |
| v3 | v1, then answer in one sentence | 4 | 42 |
| v4 | v2, then answer when the sheet has the fact | 9 | 44 |
| v5 | v1 on a bigger model | 7 | 130 |
The cells students fill in are left blank in this preview.
A fix can help one case, hurt two
Run v3 fixes the long answer, c9. It also breaks c2 and c10, which passed before: the one-sentence rule cut their figures. The total fell from 5 to 4.
A total can hide this. Compare the two runs case by case and list what was fixed and what broke.
A common mistake
Warning
Many people think a change that fixes something is a good change. Look at what it broke. Ship a change only when it breaks nothing that passed, or when you accept each break on purpose.
Write the results reader, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const fs = require(\"fs\");' '// Five recorded runs over cases c1 to c10. v2: use only figures from the sheet.' '// v3: v1 plus answer in one sentence. v4: v2 plus answer when the sheet has the fact.' '// v5: the first prompt on a bigger model. Cost is per 1000 answers.' 'const RUNS = {' ' v1: { cost: 40, kinds: \"ok ok invented ok vague ok invented refusal long ok\" },' ' v2: { cost: 42, kinds: \"ok ok ok ok vague ok ok refusal long ok\" },' ' v3: { cost: 42, kinds: \"ok vague invented ok vague ok invented refusal ok vague\" },' ' v4: { cost: 44, kinds: \"ok ok ok ok ok ok ok ok long ok\" },' ' v5: { cost: 130, kinds: \"ok ok ok ok vague ok ok refusal long ok\" },' '};' 'for (const name in RUNS) RUNS[name].kinds = RUNS[name].kinds.split(\" \");' 'fs.writeFileSync(\"runs.json\", JSON.stringify(RUNS, null, 1));' > analyse.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the results reader, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function isOk(kind) { return kind === \"ok\"; }' 'function failures(kinds) {' ' const out = {};' ' kinds.forEach(function (k) { if (isOk(k) === false) out[k] = (out[k] || 0) + 1; });' ' return out;' '}' 'const base = RUNS.v1.kinds;' 'let best = null;' >> analyse.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the results reader, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'for (const name in RUNS) {' ' const kinds = RUNS[name].kinds;' ' const fixed = [];' ' const broke = [];' ' kinds.forEach(function (k, i) {' ' if (isOk(base[i]) === false && isOk(k)) fixed.push(\"c\" + (i + 1));' ' if (isOk(base[i]) && isOk(k) === false) broke.push(\"c\" + (i + 1));' ' });' ' const passed = kinds.filter(isOk).length;' ' console.log(name, passed + \" passed\", \"cost \" + RUNS[name].cost, \"fixed [\" + fixed + \"]\", \"broke [\" + broke + \"]\", JSON.stringify(failures(kinds)));' ' if (name === \"v1\" || broke.length > 0) continue;' ' const cost = RUNS[name].cost;' ' if (best === null || passed > best.passed || (passed === best.passed && cost < best.cost)) best = { name, passed, cost };' '}' 'console.log(\"Choose: \" + best.name + \" (\" + best.passed + \" passed, cost \" + best.cost + \")\");' >> analyse.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Read the five runs
docker exec -w /home/node/project agent node analyse.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it says |
|---|---|
| v2 | 7 passed, fixed c3 and c7, broke nothing |
| v3 | 4 passed, fixed c9, broke c2 and c10 |
| v4 | 9 passed, fixed four cases, broke nothing |
| v5 | 7 passed, cost 130 |
| Last line | Choose: v4 (9 passed, cost 44) |
The cells students fill in are left blank in this preview.