Writing a post-mortem you can check
A post-mortem is a short written account of an incident: a timeline, a cause and a list of actions. It matters because one that nobody can check only records a feeling.
The cost incident at Fernside Library is over. The team writes it up, and a small script compares the draft with the log before anybody signs it.
By the end of this lesson
Note
I can write what happened, why, and what changes.
Key words
| Word | What it means | Example |
|---|---|---|
| Post-mortem | a written account of an incident and what follows | |
| Timeline | the events of an incident, in time order | |
| Cause | why it happened, in terms the log can confirm | |
| Action | a change with an owner and a due date | |
| Check | a way to see that an action worked | at most 2 attempts |
The cells students fill in are left blank in this preview.
A timeline uses the times in the log
A timeline lists what happened and when, in order. Copy each time from a record. Do not write it from memory.
A good timeline also holds the slow parts: when the cause started, when anyone noticed and when it ended. The gaps between those times are the lessons.
Worked example: the library's events
| Time | Event |
|---|---|
| 10:55 | A longer fact sheet goes live |
| 12:02 | The hourly cost alert fires |
| 12:20 | The cause is found in the trace |
| 12:35 | Attempts are capped |
| 13:00 | Cost is back under the limit |
The cells students fill in are left blank in this preview.
A cause the log can confirm
Write the cause as a sentence the log can prove or refute. It says what happened, to which requests and why.
"Every fee question was tried five times because the 40 word check could not pass" is checkable. "The system was overloaded" is not: no record can confirm it.
An action has an owner and a check
An action is a change that fixes the cause or shortens the gaps. Each one has an owner, a due date and a check: a number the log will show when the action has worked.
"Be more careful" has none of these. "At most 2 attempts a request, seen in the log within 2 days" has all three.
A common mistake
Warning
Many people think a post-mortem should say who was at fault. It should say what in the system allowed the fault. Put a name beside an action as its owner and nowhere else.
Write the draft and the log, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const fs = require(\"fs\");' 'const EVENTS = [' ' { time: \"10:55\", code: \"sheet_published\" },' ' { time: \"12:02\", code: \"alert_fired\" },' ' { time: \"12:20\", code: \"cause_found\" },' ' { time: \"12:35\", code: \"attempts_capped\" },' ' { time: \"13:00\", code: \"cost_normal\" },' '];' 'const DRAFT = {' ' timeline: [' ' { time: \"10:55\", code: \"sheet_published\" },' ' { time: \"12:02\", code: \"alert_fired\" },' ' { time: \"12:10\", code: \"cause_found\" },' ' { time: \"13:00\", code: \"cost_normal\" },' ' ],' > pm.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the draft and the log, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' cause: \"Every fee question was tried five times because the 40 word check could not pass\",' ' actions: [' ' { what: \"Cap attempts at 2\", owner: \"Sam\", check: \"The log shows at most 2 attempts a request\", days: 2 },' ' { what: \"Be more careful with the sheet\", owner: \"Sam\", check: \"\", days: 5 },' ' { what: \"Alert every 10 minutes\", owner: \"\", check: \"An alert fires within 10 minutes of a spike\", days: 60 },' ' ],' '};' 'fs.writeFileSync(\"events.json\", JSON.stringify(EVENTS, null, 1));' 'fs.writeFileSync(\"postmortem.json\", JSON.stringify(DRAFT, null, 1));' >> pm.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Create the draft files
docker exec -w /home/node/project agent node pm.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the checker, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const fs = require(\"fs\");' 'const MAX_DAYS = 30;' 'function hasDigit(text) { return /[0-9]/.test(text); }' 'function run() {' ' const events = JSON.parse(fs.readFileSync(\"events.json\", \"utf8\"));' ' const pm = JSON.parse(fs.readFileSync(\"postmortem.json\", \"utf8\"));' ' const timeline = [];' ' const actions = [];' ' let last = \"\";' ' pm.timeline.forEach(function (entry, i) {' ' const found = events.find(function (e) { return e.code === entry.code; });' ' if (found === undefined) timeline.push(i + \":unknown_event\");' ' else if ((found.time === entry.time) === false) timeline.push(i + \":wrong_time\");' ' if (entry.time < last) timeline.push(i + \":out_of_order\");' ' last = entry.time;' ' });' > check-pm.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the checker, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' events.forEach(function (e) {' ' if (pm.timeline.some(function (t) { return t.code === e.code; }) === false) timeline.push(\"missing:\" + e.code);' ' });' ' pm.actions.forEach(function (a, i) {' ' if (a.owner.trim() === \"\") actions.push(i + \":no_owner\");' ' if (a.check.trim() === \"\") actions.push(i + \":no_check\");' ' else if (hasDigit(a.check) === false) actions.push(i + \":no_number\");' ' if (a.days > MAX_DAYS) actions.push(i + \":too_late\");' ' });' ' console.log(\"Timeline:\", timeline.join(\" \"));' ' console.log(\"Actions:\", actions.join(\" \"));' ' const total = timeline.length + actions.length;' ' console.log(\"Problems:\", total);' ' console.log(total === 0 ? \"The post-mortem is ready.\" : \"The post-mortem is not ready.\");' '}' 'module.exports = { run };' 'if (require.main === module) run();' >> check-pm.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Check the draft
docker exec -w /home/node/project agent node check-pm.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the checker prints
| Line | What it says |
|---|---|
| Timeline | 2:wrong_time missing:attempts_capped |
| Actions | 1:no_check 2:no_owner 2:too_late |
| Problems | 5 |
| Last line | The post-mortem is not ready. |
The cells students fill in are left blank in this preview.
Write the fix
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const fs = require(\"fs\");' 'const pm = JSON.parse(fs.readFileSync(\"postmortem.json\", \"utf8\"));' 'pm.timeline[2].time = \"12:20\"; // the log says 12:20' 'pm.timeline.splice(3, 0, { time: \"12:35\", code: \"attempts_capped\" });' 'pm.actions[1].what = \"Test every sheet against the 40 word check before it goes out\";' 'pm.actions[1].check = \"Every eval case passes the 40 word check before a sheet is published\";' 'pm.actions[2].owner = \"Ana\";' 'pm.actions[2].days = 14;' 'fs.writeFileSync(\"postmortem.json\", JSON.stringify(pm, null, 1));' 'require(\"./check-pm\").run();' > fix-pm.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Mend the draft and check again
docker exec -w /home/node/project agent node fix-pm.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.