Scoring replies without another model
Scoring a reply with rules means checking it with ordinary code: does it parse, is the label right, is the number in range. It matters because rules are fast, free and give the same answer every time.
The shop's sorter now answers in JSON with a label, an urgency and one sentence for the customer. Each part can be checked by a rule.
By the end of this lesson
Note
I can score a reply with rules that do not need a model.
Key words
| Word | What it means | Example |
|---|---|---|
| Rule | a check written as code that a reply passes or fails | |
| Exact match | the value must equal the expected one | billing |
| Parses | the reply decodes as the format asked for | |
| Within range | a number lies between two limits | |
| Score | the number of rules a reply passes | |
| Forbidden phrase | words a reply must never contain |
The cells students fill in are left blank in this preview.
Four kinds of rule cover most replies
Most checks fit four kinds. Parses: the reply is the JSON you asked for. Exact match: a field equals the expected value. Within range: a number sits between limits. Contains: a phrase is there, or must not be.
Each rule is a few lines. None of them needs a model, so a run costs nothing.
Worked example: one rule for each part
| Part of the reply | Rule | A failing example |
|---|---|---|
| The whole reply | It parses as a JSON object | Sure, it is a delivery message |
| label | Exact match with the case | billing when delivery was wanted |
| urgency | A whole number from 1 to 3 | 7 |
| reply | Up to 120 characters, no forbidden phrase | Your refund has been approved |
The cells students fill in are left blank in this preview.
A score per case shows where it failed
Give a reply one point for each rule it passes. Do not stop at the first failure, because the points show which rule broke. A case passes only when every rule holds.
If the reply does not parse, no later rule can run, so its score is 0.
Write the model helper
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 200 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' return (await res.json()).choices[0].message.content;' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' const list = [].concat(sample);' ' const turn = messages.filter(function (m) { return m.role === \"assistant\"; }).length;' ' return list[Math.min(turn, list.length - 1)];' ' }' '}' 'module.exports = { ask };' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the rules, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const forbidden = [\"refund has been\", \"guaranteed\"];' 'function score(text, want) {' ' const out = { parses: false, label: false, range: false, reply: false };' ' let data = null;' ' try { data = JSON.parse(text); } catch (error) { return out; }' ' if (data === null || Array.isArray(data) || typeof data === \"string\" || typeof data === \"number\" || typeof data === \"boolean\") return out;' ' out.parses = true;' ' out.label = data.label === want;' > score.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the rules, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' out.range = Number.isInteger(data.urgency) && data.urgency >= 1 && data.urgency <= 3;' ' const said = typeof data.reply === \"string\" ? data.reply : \"\";' ' const low = said.toLowerCase();' ' const banned = forbidden.some(function (phrase) { return low.includes(phrase); });' ' out.reply = said.length > 0 && said.length <= 120 && banned === false;' ' return out;' '}' 'function points(out) {' ' return (out.parses ? 1 : 0) + (out.label ? 1 : 0) + (out.range ? 1 : 0) + (out.reply ? 1 : 0);' '}' 'module.exports = { score, points };' >> score.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the check script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const { score, points } = require(\"./score\");' 'function reply(label, urgency, text) {' ' return JSON.stringify({ label: label, urgency: urgency, reply: text });' '}' 'const recorded = [' ' { id: 1, want: \"billing\", text: reply(\"billing\", 2, \"Sorry about the double charge, we are checking it.\") },' ' { id: 2, want: \"delivery\", text: reply(\"billing\", 2, \"We are looking into it.\") },' ' { id: 3, want: \"other\", text: \"Sure, it is a delivery message.\" },' > check2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the check script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' { id: 8, want: \"delivery\", text: reply(\"delivery\", 7, \"Sorry to hear that.\") },' ' { id: 10, want: \"unknown\", text: reply(\"unknown\", 1, \"Your refund has been approved.\") },' '];' 'function show(name, out) {' ' console.log(name + \": \" + points(out) + \" of 4 (parses \" + out.parses + \", label \" + out.label + \", range \" + out.range + \", reply \" + out.reply + \")\");' '}' 'for (const one of recorded) show(\"Recorded case \" + one.id, score(one.text, one.want));' >> check2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the check script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const sample = reply(\"billing\", 2, \"Sorry about that, we are checking your order.\");' ' const prompt = \"Sort this shop message. Reply with only JSON that has three keys: label (billing, delivery, other or unknown), urgency (a whole number from 1 to 3) and reply (one polite sentence under 100 characters).\n\nMessage: Charged twice for one order\";' ' const text = await ask([{ role: \"user\", content: prompt }], sample, 120);' ' console.log(\"Live reply: \" + text);' ' show(\"Live case 1\", score(text, \"billing\"));' '}' 'main();' >> check2.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Score five replies and one live one
docker exec -w /home/node/project agent node check2.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Line | What it shows |
|---|---|
| Recorded case 1 | 4 of 4, every rule holds |
| Recorded case 2 | 3 of 4, the label is wrong |
| Recorded case 3 | 0 of 4, words and not JSON |
| Recorded case 8 | 3 of 4, urgency 7 is out of range |
| Recorded case 10 | 3 of 4, the forbidden phrase |
| Live case 1 | Usually 3 or 4 of 4, and it may differ each run |
The cells students fill in are left blank in this preview.
Try it
Tip
In score.js add a third forbidden phrase, sorry about the double charge, and run again. Recorded case 1 should lose its last point.
A common mistake
Warning
Many people think full marks from rules means a good reply. Rules check shape and facts only. A reply can parse, carry the right label and still be rude or unhelpful. Use rules for what code can decide, and something else for tone.