Fixed on Tuesday, and again on Friday
A post-incident review is a meeting held after an incident to learn what happened and what to change. Without one, the same mistake comes back.
A team that also measures how fast it found and fixed the problem can show whether it is getting better. You will work out those times.
By the end of this lesson
Note
I can work out the time to detect and the time to recover for an incident.
Key words
| Word | What it means |
|---|---|
| Post-incident review | a meeting to learn from an incident |
| Root cause | the deepest reason the incident could happen |
| Time to detect | minutes from the start of the attack until it is noticed |
| Time to recover | minutes from noticing until service is back |
| Average | the total divided by how many there are |
| Action item | a change with one owner and a date |
The cells students fill in are left blank in this preview.
Look for causes, not for blame
The aim of a review is to change the system, so that the next person cannot make the same slip. People speak freely only if nobody is blamed.
A good way to reach the root cause is to ask why again and again, until the answer is something you can fix.
Worked example: asking why four times
| Question | Answer |
|---|---|
| Why were customer orders lost? | Someone deleted the orders table |
| Why could they? | Their stolen password worked on the database |
| Why was the password stolen? | It was also used on a small forum |
| Why was it allowed? | No second check at sign-in. This is the root cause |
The cells students fill in are left blank in this preview.
Two clocks measure how well the team did
Write down four moments: when the attack started, when it was detected, when it was contained and when service was back. Then take away one time from another.
Time to detect runs from the start to the detection. Time to recover runs from detection to the service being back. Both are in minutes.
Worked example: one incident in minutes
| Moment | Clock | Measure | Minutes |
|---|---|---|---|
| Attack started | 09:30 | ||
| Detected | 10:45 | Detect: 10:45 minus 09:30 | 75 |
| Contained | 11:15 | Contain: 11:15 minus 10:45 | 30 |
| Service back | 13:15 | Recover: 13:15 minus 10:45 | 150 |
The cells students fill in are left blank in this preview.
Turn clock times into minutes
printf '09:30\n10:45\n' | awk -F: '{print $1 * 60 + $2}'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Work out the time to detect
awk 'BEGIN{print (10 * 60 + 45) - (9 * 60 + 30)}'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think a review is for finding who made the error. Then people hide mistakes, and the team learns less. A person did the action, but the system let them. Ask what let it happen, and change that.
An average over many incidents shows a trend
One incident tells you little. Several tell you more. Add the times and divide by how many incidents there were.
If the average time to detect falls from 90 minutes to 40, the new rules are working. If it rises, something has gone wrong.
Average three detection times
awk 'BEGIN{print (40 + 90 + 20) / 3}'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Each lesson becomes an action item
A review that ends in a list of good intentions changes nothing. Each finding becomes an action item: one change, one owner and one date.
At the next review, the team checks which items were done and whether the two times improved.
Going further
An incident was detected after 120 minutes. A review adds one new rule. The next three incidents are detected after 30, 20 and 40 minutes.
Work out the new average. Say what else you would want to know before you call the rule a success.