Coverage shows which lines never ran
Code coverage is the share of a program's lines, or of its branches, that the tests ran at least once. A team reads it to find code that no test has ever touched.
A figure of 100 percent looks like proof of quality. It is not. By the end you can say what the number tells you and what it hides.
By the end of this lesson
Note
I can read a coverage figure and say what it misses.
Key words
| Word | What it means |
|---|---|
| Coverage | the share of the code that the tests ran |
| Line coverage | the share of lines that ran at least once |
| Branch | one way an if can go: true or false |
| Branch coverage | the share of branches that ran at least once |
| Trace | a record of which lines ran |
The cells students fill in are left blank in this preview.
Line coverage counts the lines that ran
Take a function of five lines. If the tests ran three of them, line coverage is 3 out of 5, which is 60 percent.
Python has no coverage tool in its standard library that is easy to read, so you build a tiny one. It watches the program run and writes down each line number it sees.
Make a folder for this lesson
mkdir -p ~/covtestShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Go into it
cd ~/covtestShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write a function with two decisions
printf 'def stock_label(count):\n if count == 0:\n return "out"\n if count < 5:\n return "low"\n return "ok"\n' > stock.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Worked example: the lines of stock.py
| Line | Code | Runs when |
|---|---|---|
| 2 | if count == 0: | Always |
| 3 | return "out" | The count is 0 |
| 4 | if count < 5: | The count is not 0 |
| 5 | return "low" | The count is 1 to 4 |
| 6 | return "ok" | The count is 5 or more |
The cells students fill in are left blank in this preview.
Write a ten line tracer
printf 'import sys\nimport stock\n\nhit = []\n\n\ndef watch(frame, event, arg):\n if event == "line" and frame.f_code.co_name == "stock_label":\n hit.append(frame.f_lineno)\n return watch\n\n\nsys.settrace(watch)\nfor count in sys.argv[1:]:\n stock.stock_label(int(count))\nsys.settrace(None)\nprint(sorted(set(hit)))\n' > watch.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
How the tracer sees each line
sys.settrace tells Python to call watch as the program runs. watch notes each line number inside stock_label. The numbers you give on the command line are the counts to try, like the inputs of a test.
The list it prints holds the lines that ran, without repeats.
Run with one test value: 10
python3 watch.py 10Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Add a second value: 0
python3 watch.py 10 0Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Branch coverage checks both ways
An if has two branches: the condition is true, or it is false. A test that only ever makes it true can still run every line.
stock_label has two if lines, so four branches. With the values 10 and 0, line 4 only ever sees its condition false. Branch coverage notices the branch that never happened.
Worked example: four branches of stock_label
| Branch | Made to run by | Run with 10 and 0? |
|---|---|---|
| count == 0 is true | A count of 0 | Yes |
| count == 0 is false | A count such as 10 | Yes |
| count < 5 is true | A count of 1 to 4 | No |
| count < 5 is false | A count of 5 or more | Yes |
The cells students fill in are left blank in this preview.
Full coverage is not proof
Coverage says that a line ran. It does not say that anything checked the result. A test that calls a function and asserts nothing gives full coverage and proves nothing.
Coverage also cannot see a line that is missing. If the code forgets to handle a negative count, no line is uncovered.
Run the three values 10, 0 and 3
python3 watch.py 10 0 3Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think 100 percent coverage means the program has no faults. The three runs above reached every line, yet nothing compared any answer with an expected one. Use coverage to find code that is untested. Use assertions to decide whether tested code is right.