Grading the part that rules cannot reach
A model as judge is a second model call that grades the first one's output against a rubric. It matters because rules can check a label but not whether a sentence is polite or helpful.
The sorter's one-sentence reply to the customer is the case: no rule can say whether it is kind.
By the end of this lesson
Note
I can ask a model to grade a reply and keep the grading checkable.
Key words
| Word | What it means | Example |
|---|---|---|
| Judge | a model call that grades another reply | |
| Rubric | the fixed scale and what each grade means | |
| Output format | the exact shape the judge must answer in | grade and reason |
| Validate | check a reply against the format by code | |
| Recorded grade | a grade stored with its reply and reason |
The cells students fill in are left blank in this preview.
A judge needs a rubric with named levels
Do not ask a judge "is this good?". Give a rubric: a short scale where every grade says what it means. Three levels are enough for a start.
Say what to look at and what to ignore. A judge with a vague scale gives vague grades.
Worked example: a three level rubric
| Grade | Meaning |
|---|---|
| 3 | Clear, polite, and makes no promise about money |
| 2 | Acceptable, but vague or cold |
| 1 | Unhelpful, rude, or promises money |
The cells students fill in are left blank in this preview.
Fix the output format and validate every grade
Tell the judge to answer in one shape, here JSON with a grade and a short reason. Then validate it with code: it parses, the grade is a whole number from 1 to 3 and the reason is there.
A grade that fails the check is not graded. Count it. Never guess what the judge meant.
Keep the grade next to what it graded
Store each recorded grade with the reply, the reason and the rubric it used. A bare number cannot be argued with later.
Where the endpoint takes a temperature, set it low so the judge varies less. It still varies, which is why the next lesson tests it against people.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the judge script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const rubric = \"Grade the reply a shop sent to a customer from 1 to 3. 3: clear, polite, and makes no promise about money. 2: acceptable but vague or cold. 1: unhelpful, rude, or promises money. Reply with only JSON that has two keys: grade (1, 2 or 3) and reason (under 15 words).\";' > judge4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the judge script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function validate(text) {' ' let data = null;' ' try { data = JSON.parse(text); } catch (error) { return { ok: false, why: \"not JSON\" }; }' ' if (data === null || Array.isArray(data) || typeof data === \"string\" || typeof data === \"number\") return { ok: false, why: \"not an object\" };' ' if (Number.isInteger(data.grade) === false || data.grade < 1 || data.grade > 3) return { ok: false, why: \"bad grade\" };' ' const reason = typeof data.reason === \"string\" ? data.reason.trim() : \"\";' ' if (reason.length === 0 || reason.length > 100) return { ok: false, why: \"bad reason\" };' ' return { ok: true, grade: data.grade, reason: reason };' '}' >> judge4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the judge script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function judged(grade, reason) {' ' return JSON.stringify({ grade: grade, reason: reason });' '}' 'const recorded = [' ' judged(2, \"Polite but vague.\"),' ' judged(1, \"Promises money.\"),' ' \"I would give this a 3\",' ' judged(5, \"Rude.\"),' ' judged(3, \"Clear and polite, no promise.\"),' '];' 'let total = 0;' 'let valid = 0;' 'recorded.forEach(function (text, at) {' ' const result = validate(text);' ' if (result.ok) { total = total + result.grade; valid = valid + 1; }' ' console.log(\"Recorded \" + (at + 1) + \": \" + (result.ok ? \"grade \" + result.grade + \" (\" + result.reason + \")\" : \"invalid, \" + result.why));' '});' 'console.log(\"Valid: \" + valid + \" of \" + recorded.length + \". Mean grade of the valid ones: \" + (total / valid).toFixed(1));' >> judge4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the judge script, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const sample = judged(1, \"Promises money.\");' ' const text = await ask([{ role: \"user\", content: rubric + \"\n\nReply to grade: Your refund has been approved.\" }], sample, 80);' ' const result = validate(text);' ' console.log(\"Live judge said: \" + text);' ' console.log(\"Live judge: \" + (result.ok ? \"grade \" + result.grade : \"invalid, \" + result.why));' '}' 'main();' >> judge4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Validate five grades and one live one
docker exec -w /home/node/project agent node judge4.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it shows |
|---|---|
| Recorded 1 and 2 | Valid grades 2 and 1, each with a reason |
| Recorded 3 | Invalid, not JSON |
| Recorded 4 | Invalid, bad grade: 5 is off the scale |
| Recorded 5 | Valid grade 3 |
| Valid line | 3 of 5, mean grade 2.0 |
| Live judge | A grade and reason, or invalid. It may differ each run |
The cells students fill in are left blank in this preview.
Try it
Tip
In judge4.js change the reply to grade in main from the refund promise to We are looking into it. Run again and read the live grade and reason.
A common mistake
Warning
Many people trust a judge because its reasons sound sure. A fluent reason is not evidence. The judge is a model, so it can be wrong and confident. Check its format by code, and check its grades against people.