Checking a judge before you trust it
A judge is a model that grades another model's answers, an approach often called LLM-as-a-judge. It is cheap, so teams use it a lot. But a judge can lean in ways nobody sees.
The module on evals and testing shows a judge and its agreement with people. This lesson goes further, with three tests worked as numbers.
By the end of this lesson
Note
I can find out how a grader is biased before I rely on it.
Key words
| Word | What it means |
|---|---|
| Judge | a model that grades answers |
| Kappa | agreement with people beyond what chance gives |
| Position bias | a lean toward a slot, not an answer |
| Length bias | a lean toward the longer answer |
| Calibration set | answers a person graded to test a judge |
The cells students fill in are left blank in this preview.
Agreement is not enough, chance helps
Say a judge agrees with a person on 90 percent of answers. That sounds good. But if most answers pass, a judge that always says pass also scores high.
Kappa takes chance away. It runs from 1, perfect, down through 0, no better than chance.
Worked example: twenty graded answers
| Measure | Value | How it is worked out |
|---|---|---|
| Items | 20 | Counted |
| Judge and person match | 18 | 90 percent agreement |
| Chance matches, times 400 | 296 | p count times p count, plus f times f |
| Kappa | 0.62 | (360 minus 296) divided by (400 minus 296) |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think a high agreement figure proves a judge is reliable. It does not, when one label is far more common. Work out kappa, and use a calibration set with enough of each label.
Swap the order to find position bias
Show the judge two answers, X then Y, and record its pick. Then show them again as Y then X. A fair judge picks the same answer both times.
If the pick changes, the judge followed the slot. Count the changes. If they are a quarter of the cases or more, flag the judge.
Count how often the longer answer wins
Some judges like long answers. To test it, set aside pairs of equal length. Among the rest, count how often the judge picks the longer one.
About half is fine. Three in four or more means length is steering the grade.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the judge tests, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const H = \"pppppfpppppfpppfpppf\";' 'const J = \"pppppppppppfpppppppf\";' 'const rows = [' ' \"1 40 55 Y Y\", \"2 62 30 X X\", \"3 48 70 Y X\", \"4 33 51 X Y\",' ' \"5 80 45 X Y\", \"6 52 52 X X\", \"7 29 66 Y Y\", \"8 71 40 X Y\",' ' \"9 44 90 Y Y\", \"10 58 36 X X\", \"11 35 62 X Y\", \"12 77 41 X X\",' '].map(function (r) {' ' const p = r.split(\" \");' ' return { x: Number(p[1]), y: Number(p[2]), one: p[3], two: p[4] };' '});' > judge.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the judge tests, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function count(text, ch) { return text.split(ch).length - 1; }' 'const n = H.length;' 'let agree = 0;' 'for (let i = 0; i < n; i++) { if (H[i] === J[i]) agree++; }' 'const chance = count(J, \"p\") * count(H, \"p\") + count(J, \"f\") * count(H, \"f\");' 'const kappa = (agree * n - chance) / (n * n - chance);' >> judge.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the judge tests, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'let first = 0;' 'let second = 0;' 'let longer = 0;' 'let unequal = 0;' 'rows.forEach(function (r) {' ' if (r.one === \"X\" && r.two === \"Y\") first++;' ' if (r.one === \"Y\" && r.two === \"X\") second++;' ' if (r.x === r.y) return;' ' unequal++;' ' if ((r.one === \"X\") === (r.x > r.y)) longer++;' '});' >> judge.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the judge tests, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"agreement: \" + (agree * 100 / n) + \" percent, kappa: \" + kappa.toFixed(2));' ' console.log(\"swap test: \" + (first + second) + \" of \" + rows.length + \" changed (first slot \" + first + \", second slot \" + second + \")\");' ' console.log(\"longer answer won: \" + longer + \" of \" + unequal + \" (\" + Math.floor(longer * 100 / unequal) + \" percent)\");' ' const q = \"Grade this reply to Where is the library? as pass or fail, one word: Ask someone.\";' ' const reply = await ask([{ role: \"user\", content: q }], \"fail\", 5);' ' console.log(\"live grade: \" + reply.text);' '}' 'main();' >> judge.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the judge tests
docker exec -w /home/node/project agent node judge.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the tests print
| Line | What it shows |
|---|---|
| agreement: 90 percent, kappa: 0.62 | High agreement, weaker once chance is removed |
| swap test: 5 of 12 changed | Five picks followed the slot |
| longer answer won: 9 of 11 | The longer answer won 81 percent of the time |
| live grade: ... | One real grade from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.