Generated tests can pass and prove nothing
Generating tests with an AI assistant means asking it to write test cases for your code. It matters because a generated test can pass every time and still check nothing.
A set of green tests feels like safety. This lesson shows how to tell which of them could ever go red.
By the end of this lesson
Note
I can ask an assistant for tests and judge whether they test anything.
Key words
| Word | What it means |
|---|---|
| Specification | the written rule the code must follow |
| Assertion | a line that fails when the result is wrong |
| Expected value | the answer the rule gives for an input |
| Faulty copy | your code with one fault added on purpose |
| Can fail | a test that goes red when the code is wrong |
The cells students fill in are left blank in this preview.
Ask for tests from the rule
An assistant that reads only your code writes tests that match what the code does. If the code has a fault, the tests copy it.
Give it the specification instead: the words that say what the code should do. Ask for each case to be listed with its expected value, so you can check every value against the rule before you run anything.
Worked example: cases from one rule
| Price | Percent | Expected | Why this case |
|---|---|---|---|
| 995 | 10 | 896 | The discount is 99.5, rounded down to 99 |
| 500 | 0 | 500 | No discount at all |
| 1000 | 100 | 0 | Everything off |
| 10 | 50 | 5 | Exactly half |
| 0 | 50 | 0 | A free item |
The cells students fill in are left blank in this preview.
One possible request
Tip
Write tests for discount(pence, percent). The rule: take percent off pence and round the discount down to whole pence. List each case with its expected value first. Include 0 percent, 100 percent and a price of 0. Do not change the function.
A test that cannot fail proves nothing
Three shapes of test can never go red. One has no assertion, so it only shows the code ran. One compares the function with itself. One checks only the type of the answer.
The next Runs write a small function, a faulty copy that rounds the other way, and four such tests. Then one script runs every test on both.
Write the discount function
docker exec -w /home/node/project agent sh -c "mkdir -p tests-demo && printf '%s\n' '// Price in pence after a percent discount. The discount is rounded down.' 'function discount(pence, percent) {' ' return pence - Math.floor(pence * percent / 100);' '}' 'module.exports = discount;' > tests-demo/discount.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Make the faulty copy
docker exec -w /home/node/project agent sh -c "sed \"s/floor/round/\" tests-demo/discount.js > tests-demo/discount-round.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the four tests
docker exec -w /home/node/project agent sh -c "mkdir -p tests-demo && printf '%s\n' 'const assert = require(\"assert\");' '' '// Four tests, written by hand to look like ones an assistant might write.' 'module.exports = [' ' [\"returns a number\", (d) => assert.strictEqual(typeof d(995, 10), \"number\")],' ' [\"runs without an error\", (d) => { d(995, 10); }],' ' [\"agrees with itself\", (d) => assert.strictEqual(d(995, 10), d(995, 10))],' ' [\"ten percent off 995\", (d) => assert.strictEqual(d(995, 10), 896)],' '];' > tests-demo/tests.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the runner
docker exec -w /home/node/project agent sh -c "mkdir -p tests-demo && printf '%s\n' 'const tests = require(\"./tests\");' '' 'for (const file of [\"discount\", \"discount-round\"]) {' ' const d = require(\"./\" + file);' ' for (const [name, test] of tests) {' ' let result = \"pass\";' ' try {' ' test(d);' ' } catch (error) {' ' result = \"FAIL\";' ' }' ' console.log(file + \" | \" + name + \" | \" + result);' ' }' '}' > tests-demo/run.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run every test on both versions
docker exec -w /home/node/project agent node tests-demo/run.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the runner printed
| Test | Correct code | Code with one fault |
|---|---|---|
| returns a number | pass | pass |
| runs without an error | pass | pass |
| agrees with itself | pass | pass |
| ten percent off 995 | pass | FAIL |
The cells students fill in are left blank in this preview.
Break the code on purpose
Make a copy of the code with one fault, such as rounding the other way. Run the same tests on both.
A test that passes on both cannot tell them apart. Ask what fault would turn each test red. Add a test for any fault that slips through. Do this to tests an assistant wrote, and to your own.
A common mistake
Warning
Many people think more tests means better tests. Four tests passed here, and three could not fail. Count the tests that can go red, not the tests that exist.
Try it on a rule of your own
The lab has its own job, fixing the shop's fault, and this lesson leaves it alone. It is a good time to try the ideas on a different change.
Write a small new rule, such as a staff discount. Ask the assistant for tests from your written rule. Check every expected value by hand, then break a copy of the code and see which tests notice.