A bug you fix once can come back
A regression is a bug that returns after it was fixed, or a feature that worked and stops working after a change. A regression test is a test that was written for one bug, so that bug can never quietly come back.
In this lesson a customer reports a bug. You write the test first, then the fix.
By the end of this lesson
Note
I can turn a bug report into a failing test, fix the bug, and keep the test.
Key words
| Word | What it means |
|---|---|
| Regression | a bug that comes back, or a feature that stops working after a change |
| Regression test | a test written for one reported bug, kept for good |
| Bug report | what a user says went wrong, with the input that caused it |
| Red | a test run in which a test fails |
| Green | a test run in which every test passes |
| Neighbour case | an input next to the one in the report, to check the fix did not just move the bug |
The cells students fill in are left blank in this preview.
Turn a bug report into a test
A shop gives a tenth off any order of 100 pounds or more. A customer writes: "I spent exactly 100 pounds and got no discount."
That report holds the input: 10000 pence. The shop works in pence so that no price is ever a fraction. First make the folder and the code as it stands today, with the fault in it.
Make a folder for the lesson
mkdir -p ~/regtestShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Go into it
cd ~/regtestShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the shop code as it is today
printf 'def discount(total_pence):\n if total_pence > 10000:\n return total_pence // 10\n return 0\n' > shop.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the test before the fix
Turn the report into a test with exactly the customer's input. Name it after what the customer saw, so a failure reads like the report.
Run it before you fix anything. It must fail. A test that passes on the broken code would prove nothing.
Write the test from the report
printf 'import unittest\nfrom shop import discount\n\n\nclass DiscountReport(unittest.TestCase):\n def test_exactly_100_pounds_gets_a_tenth_off(self):\n self.assertEqual(discount(10000), 1000)\n' > test_shop.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run it and see it fail
python3 -m unittest test_shop 2>&1 | grep -E "^(OK|FAILED|AssertionError)"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Fix only what the test shows
The code says > where the rule says "or more". Change it to >= and nothing else. Then run the test again. It should go green.
Change the greater-than sign
sed -i 's|total_pence > 10000|total_pence >= 10000|' shop.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the test again
python3 -m unittest test_shop 2>&1 | grep -E "^(OK|FAILED|AssertionError)"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Add the neighbours of the bug
A fix can move a bug instead of removing it. Write >= 9999 by mistake and an order of 99.99 pounds gets a discount. So test the inputs on each side of the one in the report.
Worked example: the cases around the line
| Order in pence | Discount it should get | Why it is there |
|---|---|---|
| 9999 | 0 | One penny under: still no discount |
| 10000 | 1000 | The report: exactly on the line |
| 10001 | 1000 | One over: whole pence, rounded down |
| 25000 | 2500 | Far above: no cap on the discount |
The cells students fill in are left blank in this preview.
Add two neighbour tests
printf ' def test_one_penny_under_gets_nothing(self):\n self.assertEqual(discount(9999), 0)\n\n def test_over_the_line_gets_a_tenth(self):\n self.assertEqual(discount(25000), 2500)\n' >> test_shop.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run all three
python3 -m unittest test_shop 2>&1 | grep -E "^(OK|FAILED|AssertionError)"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Prove the test guards the fix
A regression test earns its place when the bug returns. Put the bug back on purpose, as a careless change would, and watch the suite catch it. Then put the fix back.
Put the bug back
sed -i 's|total_pence >= 10000|total_pence > 10000|' shop.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the tests
python3 -m unittest test_shop 2>&1 | grep -E "^(OK|FAILED|AssertionError)"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Put the fix back
sed -i 's|total_pence > 10000|total_pence >= 10000|' shop.pyShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the tests once more
python3 -m unittest test_shop 2>&1 | grep -E "^(OK|FAILED|AssertionError)"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people fix the bug, then write a test that passes, and call it done. That test was never red, so nobody knows it can catch the fault. Write it first, watch it fail, then fix.
Going further: a report with no input
Some reports have no input: "the total looked wrong on Tuesday". Before you can write a test you have to find one.
List three questions you would ask the customer to get an input you can reproduce. Which one would you ask first?