A change that breaks the feature cannot ship
An eval suite is one command. It runs your cases, scores the replies, compares with the version in production and fails when a gate breaks. It matters because a check nobody has to remember to run is the only one that protects you.
The shop wants its sorter's next prompt to be blocked automatically if it gets worse.
By the end of this lesson
Note
I can put cases, scoring, comparison and a gate into one suite.
Key words
| Word | What it means | Example |
|---|---|---|
| Eval suite | cases, scoring, comparison and a gate in one command | |
| Gate | a pass mark the run must meet | |
| Must-pass case | a case that may never fail | |
| Exit code | the number a program ends with, 0 for success | 1 |
| Report | the printed result of the run |
The cells students fill in are left blank in this preview.
A gate is a pass mark set first
Decide the gates before you see a result, or you will bend them to fit. Keep them few and clear: a minimum pass rate, a limit on the drop from the baseline and a minimum judge mean.
Add a list of must-pass cases. A refusal case that fails is a failure even when the pass rate looks fine.
Worked example: four gates for the sorter
| Gate | Limit | Why |
|---|---|---|
| Pass rate | At least 75 percent | A floor for quality |
| Drop | At most 8 points below the baseline | Catches a slide |
| Must-pass cases | Case 10 never fails | The refusal must hold |
| Judge mean | At least 2.0 | Keeps the replies kind |
The cells students fill in are left blank in this preview.
The suite reports, then exits non-zero
A report names every gate and says which failed, so the person who broke it knows what to fix. Then the script ends with an exit code: 0 when every gate holds and 1 when any fails.
A real CI service reads that number and blocks the change. The lab has no CI, so you read the code yourself.
Write the suite data, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const want = [\"billing\", \"delivery\", \"other\", \"billing\", \"delivery\", \"other\", \"billing\", \"delivery\", \"unknown\", \"unknown\", \"billing\", \"other\"];' '// The version in production passed 83 percent. This is the candidate, recorded.' 'const baseline = 83;' 'const run = [[\"billing\", 2], [\"delivery\", 2], [\"other\", 1], [\"other\", 2], [\"delivery\", 3], [\"other\", 1], [\"billing\", 2], [\"delivery\", 2], [\"unknown\", 1], [\"unknown\", 1, \"Your refund has been approved.\"], [\"billing\", 3], [\"delivery\", 2]];' > suitedata.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the suite data, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// The recorded judge grades for the candidate replies, from 1 to 3.' 'const grades = [2, 1, 3, 2, 2, 1, 3, 2, 2, 2, 3, 2];' '// The gates, written before the run. Case 10 must always pass.' 'const gates = { minRate: 75, maxDrop: 8, must: [10], minJudge: 2 };' 'module.exports = { want, baseline, run, grades, gates };' >> suitedata.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the suite, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { want, baseline, run, grades, gates } = require(\"./suitedata\");' 'function passes(entry, expected) {' ' const said = (entry[2] || \"\").toLowerCase();' ' return entry[0] === expected && entry[1] >= 1 && entry[1] <= 3 && said.includes(\"refund has been\") === false;' '}' 'const results = run.map(function (entry, at) { return passes(entry, want[at]); });' 'const rate = Math.floor(results.filter(Boolean).length * 100 / want.length);' 'const mean = grades.reduce(function (sum, grade) { return sum + grade; }, 0) / grades.length;' 'const failures = [];' > suite6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the suite, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function gate(name, ok, detail) {' ' console.log(\"Gate \" + name + \": \" + (ok ? \"ok\" : \"FAILED\") + (detail ? \" (\" + detail + \")\" : \"\"));' ' if (ok === false) failures.push(name);' '}' 'console.log(\"Cases: \" + want.length + \", pass rate: \" + rate + \" percent, baseline: \" + baseline);' 'gate(\"pass rate at least \" + gates.minRate, rate >= gates.minRate, \"\");' 'gate(\"drop at most \" + gates.maxDrop + \" points\", baseline - rate <= gates.maxDrop, \"\");' >> suite6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the suite, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const broken = gates.must.filter(function (id) { return results[id - 1] === false; });' 'gate(\"must-pass cases\", broken.length === 0, broken.length ? \"failed: \" + broken.join(\", \") : \"\");' 'gate(\"judge mean at least \" + gates.minJudge, mean >= gates.minJudge, \"mean \" + mean.toFixed(1));' 'console.log(failures.length ? \"Result: FAILED, \" + failures.length + \" gate(s)\" : \"Result: passed\");' 'process.exit(failures.length ? 1 : 0);' >> suite6.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the suite and read its exit code
docker exec -w /home/node/project agent node suite6.js; echo exit code $?Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the suite prints
| Line | What it shows |
|---|---|
| Cases | 12 cases, 75 percent, baseline 83 |
| Pass rate gate | ok, 75 meets the floor of 75 |
| Drop gate | ok, 8 points is the most allowed |
| Must-pass gate | FAILED, case 10 |
| Judge gate | ok, mean 2.1 |
| Last line | Result: FAILED, then exit code 1 |
The cells students fill in are left blank in this preview.
Try it
Tip
In suitedata.js change must from [10] to an empty list and run again. The result should pass and the exit code should be 0. Then ask yourself whether that was the right fix.
A common mistake
Warning
Many people think a gate is finished once it is written. A gate that is never reviewed drifts: cases go stale and limits stop matching what matters. Review the gates when the feature changes, and add a case after every real failure.