The job that charged a customer twice
A checkpoint is a saved record of the work already finished, so a long job can stop and start again without redoing it. Agents run for minutes or hours, and processes die.
A job charged five orders. It crashed on the third and restarted from the top. Two customers paid twice.
By the end of this lesson
Note
I can make long work safe to stop and resume.
Key words
| Word | What it means | Example |
|---|---|---|
| Checkpoint | a saved record of finished work | |
| Idempotent | safe to run twice, with one effect | |
| Resume | to continue from the last checkpoint | |
| Side effect | a change outside the program | a charge |
| Event log | a list of what happened, in order |
The cells students fill in are left blank in this preview.
A checkpoint says what finished
After each step, write the step's name to a file or a table. On start, read that record and skip what is already done.
Write a started entry before a step and a done entry after it. Then a restart can tell a step that never began from one that was cut off half way.
Worked example: where a crash can fall
| Crash falls | Record says | On resume |
|---|---|---|
| Before the step starts | Nothing about it | Run it |
| In the step, before its effect | Started | Retry it |
| After its effect, before done | Started | Retry it, so the effect repeats |
| After done | Done | Skip it |
The cells students fill in are left blank in this preview.
Write the resumable job, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const fs = require(\"fs\");' 'const CHECK = \"demo/checkpoint.json\";' 'const LOG = \"demo/charges.log\";' 'const orders = [\"A1\", \"A2\", \"A3\", \"A4\", \"A5\"];' 'const crashAt = process.argv[2];' 'fs.mkdirSync(\"demo\", { recursive: true });' 'function load() {' ' try { return JSON.parse(fs.readFileSync(CHECK, \"utf8\")); } catch (error) { return { done: [], started: null }; }' '}' 'function save(state) { fs.writeFileSync(CHECK, JSON.stringify(state)); }' 'function charge(id) {' ' const seen = fs.existsSync(LOG) ? fs.readFileSync(LOG, \"utf8\").split(\"\n\") : [];' ' if (seen.indexOf(id) >= 0) { console.log(id + \": already charged, not charging again\"); return; }' ' fs.appendFileSync(LOG, id + \"\n\");' '}' > batch1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the resumable job, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const state = load();' 'console.log(\"Done so far: \" + (state.done.join(\",\") || \"nothing\") + \". Interrupted at: \" + state.started);' 'for (const id of orders) {' ' if (state.done.indexOf(id) >= 0) { console.log(id + \": skipped, already done\"); continue; }' ' state.started = id;' ' save(state);' ' charge(id);' ' if (id === crashAt) { console.log(\"Crash while working on \" + id); process.exit(1); }' ' state.done.push(id);' ' state.started = null;' ' save(state);' ' console.log(id + \": done\");' '}' 'console.log(\"Finished. Charges: \" + fs.readFileSync(LOG, \"utf8\").trim().split(\"\n\").length);' >> batch1.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run it fresh and crash at A3
docker exec -w /home/node/project agent sh -c "rm -rf demo; node batch1.js A3"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run it again and resume
docker exec -w /home/node/project agent node batch1.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
One possible output
| Run | What it shows |
|---|---|
| First | A1 done, A2 done, then Crash while working on A3 |
| Second | Interrupted at A3, A1 and A2 skipped |
| A3 on the second run | already charged, not charging again, then done |
| Last line | Finished. Charges: 5, one for each order |
The cells students fill in are left blank in this preview.
Make each step safe to repeat
A step is idempotent when running it twice has the effect of running it once. Give each effect a key, such as the order number. Check whether the key was already used before you act.
Where an outside service supports it, send the key with the request so the service can refuse a repeat.
A common mistake
Warning
Many people think a checkpoint makes a retry safe. A checkpoint only records what finished. A step cut off after its effect is run again, so the step must be safe to repeat. Use both, a record and an idempotent step.