A report is a list of claims
A worker's report is a set of claims, and each claim says it came from a source. It matters because a worker that cannot find an answer will often write one that sounds right.
The orchestrator cannot tell the difference by reading. It has to check each claim against the source it names.
By the end of this lesson
Note
I can flag the claims in a worker's report that its sources do not support.
Key words
| Word | What it means |
|---|---|
| Claim | one statement in a report that can be true or false |
| Source | the text a claim says it came from |
| Supported | the source really says what the claim says |
| Reviewer | a second agent that reads a claim and its source |
| Spot check | reading a few claims by hand, chosen in a fixed pattern |
| Guess | a claim the worker made up to fill a gap |
The cells students fill in are left blank in this preview.
Three checks, from cheap to costly
The cheapest check is a program. Does the cited source exist, and do the numbers in the claim appear in it? It catches invented sources and wrong figures.
The next is a reviewer, a model that reads the claim and the source. The costliest is a person, who reads a few claims chosen by a fixed rule.
Worked example: what each check catches
| Check | Catches | Misses |
|---|---|---|
| Program: source and numbers | An invented source, a wrong figure | A claim that reverses the meaning |
| Reviewer model | Many reversed meanings | Its own mistakes, and a bias to agree |
| Spot check by a person | What the others both missed | Everything not picked |
The cells students fill in are left blank in this preview.
Why workers guess
A worker is rewarded for an answer, so an empty hand looks like failure. Tell it that giving no answer is allowed. The brief says to reply not found when the material does not say.
Then count the not found replies. A worker that never says not found is likely guessing.
A common mistake
Warning
Many people read a citation as proof. A claim that names a source can still be wrong about it. The check is whether the source says it, never whether the claim names one.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the claim checker, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const SOURCES = {' ' s1: \"Survey: 212 of 400 visitors asked for Sunday opening.\",' ' s2: \"Staff: six hours at 18 pounds an hour is 108 pounds each Sunday.\",' ' s3: \"Visits: Saturday had 310 visits last month, a weekday had 120.\",' '};' 'const CLAIMS = [' ' { text: \"212 of 400 visitors asked for Sunday opening.\", source: \"s1\" },' ' { text: \"Sunday staff time costs 108 pounds.\", source: \"s2\" },' ' { text: \"Visits on Saturday were 410 last month.\", source: \"s3\" },' ' { text: \"Most readers want longer loan periods.\", source: \"s1\" },' ' { text: \"Staff agree Sunday opening is fair.\", source: \"s7\" },' ' { text: \"Weekday visits were higher than Saturday visits.\", source: \"s3\" },' '];' > check.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the claim checker, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function numbers(t) { return t.match(/[0-9]+/g) || []; }' 'function words(t) {' ' return (t.toLowerCase().match(/[a-z]+/g) || []).filter(function (w) { return w.length >= 5; });' '}' >> check.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the claim checker, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// A cheap check: the source exists, its numbers match, and half the long words are there.' 'function why(c) {' ' const text = SOURCES[c.source];' ' if (text === undefined) return \"no such source\";' ' const have = numbers(text);' ' if (numbers(c.text).some(function (n) { return have.includes(n) === false; })) return \"number not in source\";' ' const w = words(c.text);' ' const found = w.filter(function (x) { return words(text).includes(x); });' ' return 2 * found.length >= w.length ? \"\" : \"words not in source\";' '}' >> check.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the claim checker, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' CLAIMS.forEach(function (c, i) {' ' console.log((i + 1) + \" \" + (why(c) === \"\" ? \"passes the check\" : \"FLAGGED: \" + why(c)));' ' });' ' console.log(\"Spot check, every 3rd claim: \" + [1, 4].join(\", \"));' ' const c = CLAIMS[5];' ' const q = [{ role: \"user\", content: \"Source: \" + SOURCES.s3 + \" Claim: \" + c.text + \" Does the source support the claim? Answer yes or no.\" }];' ' const r = await ask(q, \"no\", 10);' ' console.log(\"Reviewer on claim 6: \" + r.text.trim());' '}' 'main();' >> check.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the claim checker
docker exec -w /home/node/project agent node check.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the program cannot see
Claim 6 says weekday visits were higher than Saturday. The source says the opposite. The program passes it, because the words match. Only a reviewer or a person reads the meaning.
The one live call asks a reviewer about that claim. Its answer can differ each run, and it can be wrong.
One possible output
| Line | What it shows |
|---|---|
| 1 passes the check, 2 passes the check | Numbers and words match |
| 3 FLAGGED: number not in source | The claim says 410, the source says 310 |
| 4 FLAGGED: words not in source | Nothing in the source is about loans |
| 5 FLAGGED: no such source | It cites s7, which does not exist |
| 6 passes the check | Passed, although the meaning is reversed |
| Reviewer on claim 6: ... | A second opinion, never proof |
The cells students fill in are left blank in this preview.