Generated code is untrusted until it is tested
Testing generated code means running it on inputs you chose, against answers you worked out yourself, before you rely on it. It matters because an assistant that writes both the code and its tests can write tests that simply agree with the code.
A passing test only means something when a person decided what it should expect.
By the end of this lesson
Note
I can test generated code before using it.
Key words
| Word | What it means |
|---|---|
| Test case | one input with the answer you expect |
| Expected value | the answer worked out from the rule |
| Edge case | an unusual input where code often fails |
| Boundary | the value where a rule changes |
| Regression | an old behaviour that breaks after a change |
The cells students fill in are left blank in this preview.
Work out the expected answer before you run
Take the rule, not the code, and work out the answer by hand. Write it down. Then run the code.
If you read the answer off the code, the test can never disagree with it, so it can never find a fault. The rule is the judge.
Worked example: cases chosen from a rule
| Age | Expected price | Why this case |
|---|---|---|
| 4 | 0 | The last age that is free |
| 5 | 500 | The first age that pays the child price |
| 15 | 500 | The last child age |
| 16 | 1000 | The first adult age |
| 65 | 600 | The first senior age |
The cells students fill in are left blank in this preview.
The rule behind that table
Rule: free under 5, 500 pence from 5 to 15, 1000 pence from 16 to 64, 600 pence from 65.
A boundary is where the price changes. Faults hide on both sides of a boundary, so each boundary gets two cases: the last value before it and the first value after it.
Edge case: no money at all
docker exec -w /home/node/project agent node -e "console.log(require('./shop').asPounds(0))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Edge case: under ten pence
docker exec -w /home/node/project agent node -e "console.log(require('./shop').asPounds(5))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Edge case: an item not in the shop
docker exec -w /home/node/project agent node -e "console.log(require('./shop').total(['cake']))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Two pass, the third needs a decision
The first Run printed £0.00 and the second £0.05. Both look right: the shop formats pence as pounds with two decimal places.
The third printed undefined. A basket with an item that has no price gives no error and no message, only a value that will break something later. A test for this case forces a decision: refuse it, or charge nothing.
Decimal fractions are an edge case
Computers store most decimal fractions as close approximations. The next two Runs add 0.1 and 0.2 and then ask whether the answer is exactly 0.3.
This is why the tuck shop keeps its prices in whole pence. Generated code that adds money as decimals needs a test with prices that expose it.
Add 0.1 and 0.2
docker exec agent node -e "console.log(0.1 + 0.2)"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Ask whether the answer equals 0.3
docker exec agent node -e "console.log(0.1 + 0.2 === 0.3)"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What those two lines printed
The first printed 0.30000000000000004 and the second printed false. The sum is a hair above 0.3, so an exact comparison fails.
A test that adds prices in decimals would have found this. A test with only whole numbers would have missed it.
A test that cannot fail proves nothing
After your tests pass, break the code on purpose. Change >= to > and run them again. If they still pass, they do not test that boundary.
This is the check to use on tests an assistant wrote for you. Ask of each one: what fault would make this fail?
A common mistake
Warning
Many people think that if the assistant's own tests pass, the code is correct. The same program may have written both, so they can share one wrong idea. Add a case you worked out yourself from the rule.