How to judge an evaluation set
An evaluation set is the list of cases you use to score an AI feature. A score is only as good as that list.
The module on evals and testing shows how to build a first set. This lesson asks a harder question: can you trust the score it gives?
By the end of this lesson
Note
I can say whether an evaluation set is good enough to trust.
Key words
| Word | What it means | Example |
|---|---|---|
| Coverage | how well the cases spread over the kinds of input | |
| Label | the answer a person says is right for a case | yes |
| Disagreement | two people giving one case two labels | |
| Margin | how far a score may move by chance | |
| Gap | the difference between two versions' scores |
The cells students fill in are left blank in this preview.
Coverage means every kind is tested
Sort your cases into kinds, such as plain, mixed, edge and refusal. Count each kind. A kind with only one or two cases is thin.
A thin kind hides failures. If a feature fails every refusal case, two cases out of twenty move the score very little.
Worked example: twenty cases by kind
| Kind | Cases | Share | Verdict |
|---|---|---|---|
| plain | 10 | 50 percent | Enough |
| mixed | 5 | 25 percent | Enough |
| edge | 3 | 15 percent | Enough |
| refusal | 2 | 10 percent | Thin, add more |
The cells students fill in are left blank in this preview.
Two people check each other's labels
A label is a person's opinion. Ask a second person to label the same cases without seeing the first labels. Then compare.
Where they match, the case is clear. Where they differ, the case or the rule is unclear. Review those cases before you score anything.
A common mistake
Warning
Many people think a label from one expert is the truth. It is one opinion. Use it only when a second person agrees, or when you have written down why the case has that label.
A small set cannot tell two versions apart
A score moves by chance. A rough rule for the margin is 100 divided by the square root of the number of cases, in points. With 100 cases, the margin is 10 points.
A gap smaller than the margin may be noise. To see a gap of 5 points you need about 400 cases.
Worked example: cases and margin
| Cases | Margin in points | Smallest gap you can trust |
|---|---|---|
| 25 | 20 | About 20 points |
| 100 | 10 | About 10 points |
| 400 | 5 | About 5 points |
| 2,500 | 2 | About 2 points |
The cells students fill in are left blank in this preview.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the set audit, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const rows = [' ' \"1 plain yes yes\", \"2 plain yes yes\", \"3 plain no no\", \"4 plain yes yes\", \"5 plain no no\",' ' \"6 plain yes yes\", \"7 mixed yes no\", \"8 mixed no no\", \"9 mixed yes yes\", \"10 plain no no\",' ' \"11 mixed no no\", \"12 edge yes yes\", \"13 mixed yes no\", \"14 edge no no\", \"15 plain yes yes\",' ' \"16 plain no no\", \"17 edge yes yes\", \"18 plain yes no\", \"19 refusal no no\", \"20 refusal no no\",' '].map(function (r) {' ' const p = r.split(\" \");' ' return { id: Number(p[0]), kind: p[1], a: p[2], b: p[3] };' '});' > audit.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the set audit, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const counts = {};' 'rows.forEach(function (r) { counts[r.kind] = (counts[r.kind] || 0) + 1; });' 'const thin = Object.keys(counts).filter(function (k) { return counts[k] < 3; });' 'const apart = rows.filter(function (r) { return r.a < r.b || r.a > r.b; });' 'const same = rows.length - apart.length;' 'const gap = 5;' 'const needed = (100 / gap) * (100 / gap);' >> audit.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the set audit, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"cases: \" + rows.length + \" by kind: \" + JSON.stringify(counts));' ' console.log(\"thin kinds (under 3): \" + thin.join(\", \"));' ' console.log(\"agree: \" + (same * 100 / rows.length) + \" percent\");' ' console.log(\"review these ids: \" + apart.map(function (r) { return r.id; }).join(\", \"));' ' console.log(\"to see a gap of \" + gap + \" points you need about \" + needed + \" cases\");' ' const q = \"In one short sentence, why should two people label the same cases?\";' ' const sample = \"Where they disagree, the case or the rule is unclear, and a score on it means little.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> audit.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the audit
docker exec -w /home/node/project agent node audit.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the audit prints
| Line | What it shows |
|---|---|
| cases: 20 by kind: {...} | How many cases each kind has |
| thin kinds (under 3): refusal | Kinds that need more cases |
| agree: 85 percent | How often the two people matched |
| review these ids: 7, 13, 18 | Cases where they differ |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.