A judge that nobody has checked
Testing a judge means comparing its grades with grades a person gave to the same replies. It matters because a judge is a model, and a model can lean toward the first answer or the longest one without telling you.
The shop now has judge grades for its replies. Before it trusts them, it needs to know how far they agree with people.
By the end of this lesson
Note
I can test a judge against grades a person gave.
Key words
| Word | What it means |
|---|---|
| Calibration set | replies a person graded to test the judge |
| Agreement | the share of grades the judge and person match on |
| Position bias | a lean toward the first or the second slot |
| Length bias | a lean toward longer replies |
| Swap | judge a pair again with the two replies exchanged |
The cells students fill in are left blank in this preview.
Test the judge on a calibration set
Pick ten to thirty replies. A person grades them with the same rubric, without seeing the judge's grades. Then the judge grades the same replies.
Agreement is the share of replies where the two grades match. Also count how many are within one grade, because 2 against 3 is a small miss and 1 against 3 is a large one.
Worked example: five replies, two sets of grades
| Reply | Person | Judge | Match |
|---|---|---|---|
| 1 | 3 | 3 | exact |
| 2 | 1 | 1 | exact |
| 3 | 2 | 3 | within one |
| 4 | 1 | 3 | miss by two |
| 5 | 2 | 2 | exact |
The cells students fill in are left blank in this preview.
Two biases to look for
Position bias: shown two replies, the judge tends to pick the first slot. Test it by a swap. Ask again with the replies exchanged. A fair judge picks the other slot the second time.
Length bias: the judge gives longer replies higher grades than people do. Compare its average gap to the people on long replies and on short ones.
Write the calibration data
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// A calibration set: ten replies a person graded 1 to 3, then the judge graded the same ten.' 'const human = [3, 1, 2, 2, 1, 2, 3, 1, 1, 2];' 'const judge = [3, 1, 2, 3, 2, 2, 3, 3, 1, 3];' 'const words = [14, 5, 6, 16, 4, 5, 17, 12, 6, 15];' '// Six pairs, each judged twice with the two replies swapped. The slot the judge chose each time.' 'const first = [\"A\", \"A\", \"A\", \"B\", \"A\", \"A\"];' 'const second = [\"B\", \"A\", \"A\", \"A\", \"A\", \"B\"];' 'function percent(count, total) {' ' return Math.floor(count * 100 / total);' '}' 'module.exports = { human, judge, words, first, second, percent };' > calib.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the measuring script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { human, judge, words, first, second, percent } = require(\"./calib\");' 'const longest = 12;' 'let exact = 0;' 'let near = 0;' 'let longGap = 0;' 'let longCount = 0;' 'let shortGap = 0;' 'let shortCount = 0;' 'for (let at = 0; at < human.length; at = at + 1) {' ' const gap = judge[at] - human[at];' ' if (gap === 0) exact = exact + 1;' ' if (Math.abs(gap) <= 1) near = near + 1;' ' if (words[at] >= longest) { longGap = longGap + gap; longCount = longCount + 1; }' ' else { shortGap = shortGap + gap; shortCount = shortCount + 1; }' '}' > measure5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the measuring script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'let same = 0;' 'for (let at = 0; at < first.length; at = at + 1) {' ' if (first[at] === second[at]) same = same + 1;' '}' 'console.log(\"Agreement: \" + percent(exact, human.length) + \" percent exact, \" + percent(near, human.length) + \" percent within one\");' 'console.log(\"Average gap on long replies: \" + (longGap / longCount).toFixed(1) + \", on short ones: \" + (shortGap / shortCount).toFixed(1));' 'console.log(\"Same slot both times: \" + same + \" of \" + first.length);' 'const flags = [];' 'if (percent(exact, human.length) < 70) flags.push(\"low agreement\");' 'if (percent(same, first.length) > 30) flags.push(\"position bias\");' 'if (longGap / longCount > 0.5) flags.push(\"length bias\");' 'console.log(\"Flags: \" + (flags.length ? flags.join(\", \") : \"none\"));' >> measure5.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Measure the judge
docker exec -w /home/node/project agent node measure5.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the measurement prints
| Line | What it shows |
|---|---|
| Agreement | 60 percent exact, 90 percent within one |
| Average gap | 0.8 on long replies, 0.2 on short ones |
| Same slot | 3 of 6 pairs, a lean toward the first slot |
| Flags | low agreement, position bias, length bias |
The cells students fill in are left blank in this preview.
Try it
Tip
In calib.js change the second list to the opposite of the first, which is B, B, B, A, B, B. Run again. The position bias flag should go.
A common mistake
Warning
Many people think a judge that agrees with people most of the time can be trusted everywhere. Agreement holds only for cases like the calibration set. Measure it again after any change to the rubric, the judge or the kind of reply.