A checkout that charged twice
Software testing means running a program on purpose, with known inputs, to see whether it gives the answers it should. You do it to find faults before a customer does.
A shop's checkout once charged some customers twice. Nobody noticed until the complaints came in. In this module you learn to write the tests that would have caught it.
By the end of this lesson
Note
I can say which kind of test finds which kind of fault.
Key words
| Word | What it means | Example |
|---|---|---|
| Fault | a mistake in the code | len(n) - 1 |
| Failure | a wrong result that a user or a test sees | |
| Test case | one input and the answer you expect | |
| Unit test | a test of one small part on its own | |
| Integration test | a test of two or more parts working together | |
| System test | a test of the whole program, as a user would use it |
The cells students fill in are left blank in this preview.
Fault and failure are not the same
A fault is a mistake somebody typed into the code. A failure is the wrong result that the fault causes when the code runs.
The two are not the same. A fault can sit in a program for years and cause no failure, because nobody ever runs that line with the input that shows it.
Run a mean with a fault in it
python3 -c "n = [2, 4, 6]; print(sum(n) / (len(n) - 1))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the wrong mean tells you
The Run printed 6.0, but the mean of 2, 4 and 6 is 4. The failure is the wrong 6.0 on the screen.
The fault is the - 1. The mean divides the sum by the count of numbers, which is len(n), and not by one less than that.
The same fault, one number
python3 -c "n = [5]; print(sum(n) / (len(n) - 1))"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A test states the answer, then checks
A test case is an input and the answer you expect. The test runs the code and compares what it got with what you expected. If they differ, the test fails.
Python has a word for this: assert. It does nothing when its condition is true, and stops with an AssertionError when it is false.
Let assert catch the wrong mean
python3 -c "n = [2, 4, 6]; assert sum(n) / (len(n) - 1) == 4"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Unit, integration and system tests
A unit test checks one small part, such as one function, with nothing else involved. It runs in a fraction of a second.
An integration test checks that two or more parts work together, for example the basket and the price list.
A system test drives the whole program the way a user does, from the screen to the database. It is the slowest.
Worked example: one fault, three tests
| Test level | What the test does | What it finds in the checkout |
|---|---|---|
| Unit | Calls total(prices) with [250, 199] | The sum adds 250 and 199 wrongly |
| Integration | Passes a basket through the price list and the total | The price list gives pence and the total expects pounds |
| System | Clicks Pay in a browser with a full basket | The Pay button sends the order twice |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think that when every test passes, the program is correct. A passing test shows only that this input gave the right answer. Testing can show that a fault exists. It can never show that there are none. Call a passing suite "no fault found yet".
Late faults cost more to fix
A fault found by a unit test is fixed in minutes, by the person who just wrote the line. The same fault found by a customer needs a bug report, a search for the cause, a fix, a new release and perhaps a refund.
That is why you test early and often. The table gives an example, with made-up hours. Your own numbers will differ, but the shape will not.
Worked example: hours to fix one fault
| Where the fault is found | Hours to fix | Why |
|---|---|---|
| A unit test, minutes after writing it | 0.1 | The code is still in your head |
| An integration test, a day later | 1 | You must find which part is wrong |
| A system test before release | 4 | The whole team waits for the fix |
| A customer report after release | 20 | Support, a fix, a release and refunds |
The cells students fill in are left blank in this preview.