A test the model has already seen
Train, validation and test are three separate piles of examples. The model learns from the first, you tune with the second and you judge with the third.
If a test case also sits in the training pile, the model may simply remember it. The score then looks good and means nothing.
By the end of this lesson
Note
I can keep test cases out of training data.
Key words
| Word | What it means |
|---|---|
| Train set | the examples the model learns from |
| Validation set | examples used to tune choices |
| Test set | examples kept back to judge the result |
| Leak | a test case that also sits in training |
| Duplicate | an example that appears twice |
The cells students fill in are left blank in this preview.
Three piles with three jobs
Split your checked file before you train. Most examples go to the train set. A smaller pile, the validation set, lets you compare choices as you go. The test set is touched once, at the end.
Split by a fixed rule, such as position, so the same rows land in the same pile every time.
Worked example: ten rows, three piles
| Pile | Rows | Used for |
|---|---|---|
| Train | 1 to 6 | Learning |
| Validation | 7 and 8 | Choosing between versions |
| Test | 9 and 10 | One final judgement |
The cells students fill in are left blank in this preview.
A near copy is still a leak
Row 1 says "Where is my parcel". Row 7 says "Where is my parcel?". The words are the same, so the model has seen the answer.
To catch these, make both texts plain first. Lower the case, drop the marks and squeeze the spaces. Then compare.
A common mistake
Warning
Many people think shuffling the file at random keeps the test set safe. A shuffle moves copies of the same example into different piles. Remove the duplicates first, then split.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the leak finder, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const rows = [' ' \"Where is my parcel\", \"I was charged twice\", \"Please cancel my order\",' ' \"The box arrived crushed\", \"Wrong invoice amount\", \"Can I change my address\",' ' \"Where is my parcel?\", \"Love your shop\", \"My refund has not come\", \"Is the shop open on Sunday\",' '];' > adapt3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the leak finder, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function normal(text) {' ' let out = \"\";' ' for (const ch of text.toLowerCase()) {' ' if (/[a-z0-9 ]/.test(ch)) out = out + ch;' ' }' ' return out.split(\" \").filter(function (w) { return w.length > 0; }).join(\" \");' '}' 'const train = rows.slice(0, 6);' 'const validation = rows.slice(6, 8);' 'const test = rows.slice(8);' >> adapt3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the leak finder, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function leaks(from, into) {' ' const seen = from.map(normal);' ' return into.filter(function (t) { return seen.includes(normal(t)); });' '}' 'async function main() {' ' console.log(\"train \" + train.length + \", validation \" + validation.length + \", test \" + test.length);' ' console.log(\"leaks into validation: \" + leaks(train, validation).join(\" | \"));' ' console.log(\"leaks into test: \" + leaks(train, test).length);' ' const q = \"In one sentence, why must a test case never appear in training data?\";' ' const sample = \"A model can repeat an example it has seen, so a leaked test case makes the score look better than it is.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 40);' ' console.log(\"model: \" + reply.text);' '}' 'main();' >> adapt3.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the leak finder
docker exec -w /home/node/project agent node adapt3.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the finder reported
The Run splits ten invented sentences by position and compares the plain text of each. It finds one leak, into the validation pile, and none in the test pile.
The last line is one short model call, or a Recorded sample.