Passing one test is not a fix
Debugging with an AI assistant means giving it what it needs to find a fault, then checking the fix it suggests. It matters because an assistant will offer a change that turns your failing test green and breaks something else.
You decide whether a fix explains the failure.
By the end of this lesson
Note
I can give an assistant what it needs to find a bug and check its fix.
Key words
| Word | What it means |
|---|---|
| Failing test | a test that shows the fault by going red |
| Error message | the text a program prints when it goes wrong |
| Reproduce | to make the fault happen again on demand |
| Candidate fix | a change that might remove the fault |
| Held-back case | an input you keep to check a fix with |
The cells students fill in are left blank in this preview.
Give the assistant the facts
Say what you know. Give the failing test and the exact error message, not a summary. Include the smallest piece of code that shows the fault, so the assistant can reproduce it.
Say what you already tried and what you ruled out. Then ask for an explanation before any change.
One possible request
Tip
This test fails: boxes(10, 4) should give 3 and gives 2. Here is the function and the test. I checked that boxes(8, 4) is right. Explain why it fails before you change anything, then suggest one change.
Write a function with a fault
docker exec -w /home/node/project agent sh -c "mkdir -p debug-demo && printf '%s\n' '// How many boxes of this size are needed for this many items.' 'function boxes(items, size) {' ' return Math.floor(items / size);' '}' 'module.exports = boxes;' > debug-demo/pack.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the test
docker exec -w /home/node/project agent sh -c "mkdir -p debug-demo && printf '%s\n' 'const boxes = require(\"./pack\");' 'const tests = [[4, 4, 1], [8, 4, 2], [10, 4, 3]];' 'let passed = 0;' 'for (const [items, size, expected] of tests) {' ' const got = boxes(items, size);' ' if (got === expected) {' ' passed = passed + 1;' ' } else {' ' console.log(\"FAIL boxes(\" + items + \", \" + size + \"): expected \" + expected + \", got \" + got);' ' }' '}' 'console.log(passed + \" of \" + tests.length + \" tests pass\");' > debug-demo/test.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the test and read the failure
docker exec -w /home/node/project agent node debug-demo/test.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Try one change, then check it
Apply one candidate fix and run every test, not only the one that failed. Then run a held-back case: an input the assistant never saw.
The next Runs try four candidate fixes for the function above. They are written by hand to stand for the kinds a fix can be.
Write the four candidate fixes, part 1
docker exec -w /home/node/project agent sh -c "mkdir -p debug-demo && printf '%s\n' 'const fixes = [' ' [\"round up\", (i, s) => Math.ceil(i / s)],' ' [\"round to nearest\", (i, s) => Math.round(i / s)],' ' [\"special case\", (i, s) => (i === 10 ? 3 : Math.floor(i / s))],' ' [\"add the size first\", (i, s) => Math.floor((i + s) / s)],' '];' > debug-demo/try.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the four candidate fixes, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const tests = [[4, 4, 1], [8, 4, 2], [10, 4, 3]];' 'const heldBack = [[7, 3, 3]];' 'const count = (f, set) => set.filter(([i, s, e]) => f(i, s) === e).length;' '' 'for (const [name, f] of fixes) {' ' const held = count(f, heldBack) === 1 ? \"pass\" : \"FAIL\";' ' console.log(name + \": \" + count(f, tests) + \" of 3 tests, held-back case \" + held);' '}' >> debug-demo/try.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Try each fix on the tests
docker exec -w /home/node/project agent node debug-demo/try.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What each fix did
| Fix | Tests passed | Held-back case |
|---|---|---|
| round up | 3 of 3 | pass |
| round to nearest | 3 of 3 | FAIL |
| special case | 3 of 3 | FAIL |
| add the size first | 1 of 3 | pass |
The cells students fill in are left blank in this preview.
A good fix explains the failure
The function rounds down, so a box that is only part full is lost. 10 items in boxes of 4 need 3 boxes, and the code gives 2. Rounding up explains that.
Rounding to nearest only happened to give 3 for 2.5. A special case for 10 explains nothing. Ask the assistant: why does your change fix the cause?
A common mistake
Warning
Many people think a green test means the bug is fixed. A fix can pass the test that failed and still be wrong for other inputs. Check it on a case the assistant did not see.
Try it on a bug of your own
The lab's own job is the shop's fault, and this lesson leaves it alone. It is a good time to try the ideas on a different bug.
Write a small function with a fault of your own. Give an assistant the failing test and the exact message. Run its fix on a held-back case, then ask why it works.