A failure in use makes the best test
An eval from production turns a real failure into a test case. It matters because customers find faults no test author thought of, but their messages hold names, addresses and numbers you must not keep.
The shop agent fails four times this week. Which parts of those chats may go into your test set?
By the end of this lesson
Note
I can turn a failure in use into a case without keeping what should not be kept.
Key words
| Word | What it means |
|---|---|
| Trace | the record of one real run |
| Redaction | replacing personal data with a marker |
| Sampling | choosing which traces to look at |
| Control case | a passing trace kept to show what must not break |
| Drift | a set that changes until it no longer measures the same thing |
The cells students fill in are left blank in this preview.
Choose which traces to read
You cannot read every trace. Sampling picks. Take every failure a user reported or a check flagged. Add a few passing traces as control cases, so a fix cannot break what worked.
A failure is a case only once a person has written what should have happened.
Worked example: six traces become four cases
| Trace | Outcome | What happens to it |
|---|---|---|
| t1 | failed | a case, with the email replaced |
| t2 | ok | a control case |
| t3 | failed | a case, with the phone number replaced |
| t4 | failed | a case |
| t5 | failed | dropped: the same input as t4 once redacted |
| t6 | ok | not sampled |
The cells students fill in are left blank in this preview.
Remove personal data before you store a case
Redaction replaces each name, address and number with a marker such as [email]. Do it by rules you can read and test, never by asking a model, and redact the expected text too.
Then check. Search the finished cases for what should be gone. A case that still holds an address is a data leak in your test set.
Keep the set from drifting
Drift means the set quietly changes until a score from last month no longer means the same thing. Give the set a version. Add cases only by review, never by a script that runs every night.
Put a limit on its size and drop duplicates, so a flood of one fault does not outweigh the rest.
Write the recorded traces
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// Invented traces from a shop agent. The people and numbers are made up.' 'const traces = [' ' { id: \"t1\", outcome: \"failed\", user: \"Refund order A17, my email is [email protected]\", expected: \"Ask which item, then refund once\" },' ' { id: \"t2\", outcome: \"ok\", user: \"Where is order B02?\", expected: \"Say it shipped\" },' ' { id: \"t3\", outcome: \"failed\", user: \"Call me on +44-7700-900123 about order C11\", expected: \"Offer the help page, promise no call\" },' ' { id: \"t4\", outcome: \"failed\", user: \"Where is my parcel? [email protected]\", expected: \"Look up the order, then ask for the number\" },' ' { id: \"t5\", outcome: \"failed\", user: \"Where is my parcel? [email protected]\", expected: \"Look up the order, then ask for the number\" },' ' { id: \"t6\", outcome: \"ok\", user: \"Hello\", expected: \"Greet the customer\" },' '];' 'module.exports = { traces };' > traces.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the case builder, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { traces } = require(\"./traces\");' 'function redact(text) {' ' return text.split(\" \").map(function (word) {' ' let core = word;' ' let tail = \"\";' ' while (core.length > 0 && \".,?\".includes(core[core.length - 1])) {' ' tail = core[core.length - 1] + tail;' ' core = core.slice(0, -1);' ' }' ' if (core.includes(\"@\")) return \"[email]\" + tail;' ' const digits = core.split(\"\").filter(function (c) { return c >= \"0\" && c <= \"9\"; }).length;' ' const plain = core.split(\"\").every(function (c) { return (c >= \"0\" && c <= \"9\") || c === \"+\" || c === \"-\"; });' ' if (plain && digits >= 7) return \"[phone]\" + tail;' ' return word;' ' }).join(\" \");' '}' > cases.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the case builder, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const seen = [];' 'const cases = [];' 'const dropped = [];' 'let okSeen = 0;' 'traces.forEach(function (trace) {' ' const made = { input: redact(trace.user), expected: redact(trace.expected), source: trace.id };' ' if (trace.outcome === \"ok\") {' ' if (okSeen % 2 === 0) { made.kind = \"control\"; cases.push(made); }' ' okSeen = okSeen + 1;' ' return;' ' }' ' if (seen.includes(made.input)) { dropped.push(trace.id); return; }' ' seen.push(made.input);' ' made.kind = \"failure\";' ' cases.push(made);' '});' >> cases.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the case builder, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'cases.forEach(function (one, at) {' ' console.log(\"Case \" + (at + 1) + \" (\" + one.kind + \", from \" + one.source + \"): \" + one.input);' '});' 'console.log(\"Dropped as a duplicate: \" + dropped.join(\", \"));' 'const left = cases.filter(function (one) { return one.input.includes(\"@\") || one.input.includes(\"7700\"); });' 'console.log(\"Cases still holding personal data: \" + left.length);' 'console.log(\"Set version 1 with \" + cases.length + \" cases\");' >> cases.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Build cases from the traces
docker exec -w /home/node/project agent node cases.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the builder prints
| Line | What it shows |
|---|---|
| Cases 1 to 4 | three failures and one control, personal data replaced |
| Dropped | t5 |
| Personal data left | 0 |
| Set | version 1 with 4 cases |
The cells students fill in are left blank in this preview.
Try it
Tip
In traces.js add a failed trace whose user text holds an address such as [email protected], and run again. The case should show [email]. Then take the redact calls out of cases.js and see what leaks.
A common mistake
Warning
Many people redact only the user's message. The expected text, the trace id and the tool arguments can hold the same address. Redact every field you keep, or keep only the fields you need, and test the finished set for what should be gone.