Generating test cases without cheating
Synthetic evaluation data is test cases made by a program. Writing a thousand cases by hand is slow, so teams generate them from templates.
Made-up cases can repeat, or give away their own answers. This lesson makes cases from a seed and removes the bad ones.
By the end of this lesson
Note
I can make new cases from a seed and say they are not copies.
Key words
| Word | What it means | Example |
|---|---|---|
| Template | a sentence with gaps to fill | Swap the {item} |
| Seed | a start number that makes a generator repeat | |
| Near duplicate | a case that is almost the same as another | |
| Leak | a case that gives away its answer or its training text | |
| Similarity | how many words two texts share, as a percent |
The cells students fill in are left blank in this preview.
A template and a seed make cases
A template is a sentence with gaps, such as Swap the {item}. A list of items fills the gap. Each template has a label, the answer a good feature should give.
A seed is a number that starts a counter. The same seed always picks the same templates and items, so the cases repeat.
Worked example: a seeded generator
| Step | What happens |
|---|---|
| Start | The seed is the first number |
| Draw | Multiply by 16807, keep the rest after dividing by 2147483647 |
| Pick | That number divided by the list size, take the rest |
| Fill | Put the picked item in the gap |
| Repeat | The same seed gives the same cases every time |
The cells students fill in are left blank in this preview.
Near duplicates make a set look bigger
Two cases that differ by one word teach nothing new. Count the words they share. Divide by all the different words in both texts.
Say 80 percent or more is a near duplicate. Keep the first case and reject the later one.
A common mistake
Warning
Many people think more generated cases always means a better test. If most are copies, the set is smaller than it looks. Count only the cases that are not near duplicates.
Two kinds of leak spoil a test
A leak of the answer is a case that holds its own label, such as Please refund my lamp with the label refund. A feature can pass by spotting the word.
A leak of training text is a case that matches text the feature has already seen. A good score on it proves memory, not skill.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the generator, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const templates = [' ' [\"My {item} arrived {fault}, I want my money back\", \"refund\"],' ' [\"Please refund my {item}, it was {fault}\", \"refund\"],' ' [\"Where is my {item}? It has not arrived\", \"tracking\"],' ' [\"Can I swap my {item} for a different colour\", \"exchange\"],' ' [\"The {item} is {fault} and I would like my money returned\", \"refund\"],' ' [\"Order status for my {item} please\", \"tracking\"],' '];' 'const items = [\"kettle\", \"lamp\", \"scarf\", \"mug\"];' 'const faults = [\"broken\", \"damaged\", \"cracked\"];' 'const train = [\"Where is my kettle? It has not arrived\", \"Can I swap my lamp for a different colour\"];' > synth.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the generator, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'let s = 23;' 'function next(size) { s = (s * 16807) % 2147483647; return s % size; }' 'function generate(n) {' ' const out = [];' ' for (let i = 1; i <= n; i++) {' ' const t = templates[next(templates.length)];' ' let text = t[0];' ' if (text.indexOf(\"{item}\") >= 0) text = text.replace(\"{item}\", items[next(items.length)]);' ' if (text.indexOf(\"{fault}\") >= 0) text = text.replace(\"{fault}\", faults[next(faults.length)]);' ' out.push({ id: i, text: text, label: t[1] });' ' }' ' return out;' '}' >> synth.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the generator, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function wordList(text) {' ' const w = text.toLowerCase().replace(/[,?.]/g, \" \").split(\" \").filter(function (x) { return x.length > 0; });' ' return w.filter(function (x, i) { return w.indexOf(x) === i; });' '}' 'function similarity(a, b) {' ' const wa = wordList(a);' ' const wb = wordList(b);' ' const shared = wa.filter(function (w) { return wb.indexOf(w) >= 0; }).length;' ' return Math.floor(shared * 100 / (wa.length + wb.length - shared));' '}' 'function near(text, list) {' ' return list.some(function (t) { return similarity(text, t) >= 80; });' '}' >> synth.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the generator, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const kept = [];' ' const reasons = {};' ' generate(12).forEach(function (c) {' ' let why = \"\";' ' if (wordList(c.text).indexOf(c.label) >= 0) why = \"leaks answer\";' ' else if (near(c.text, train)) why = \"leaks training text\";' ' else if (near(c.text, kept.map(function (k) { return k.text; }))) why = \"duplicate\";' ' if (why === \"\") kept.push(c);' ' else reasons[why] = (reasons[why] || 0) + 1;' ' });' >> synth.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the generator, part 5
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' console.log(\"kept \" + kept.length + \" of 12\");' ' console.log(\"rejected: \" + JSON.stringify(reasons));' ' console.log(\"first kept: \" + kept[0].text + \" -> \" + kept[0].label);' ' const sample = \"Fix the template so the label word is never in the text.\";' ' const q = \"In one short sentence, how do you stop a made-up case from giving away its answer?\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> synth.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the generator
docker exec -w /home/node/project agent node synth.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| kept 7 of 12 | Cases left after the checks |
| rejected: {...} | How many were thrown out, and why |
| first kept: ... -> ... | One clean case and its label |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
The cells students fill in are left blank in this preview.