Saturday, 7pm
On Saturday evening the payment page starts showing an error. Three engineers notice at once.
Each starts fixing it alone. Two of them change the same server. Nobody tells customers or the support team.
The outage lasts two hours. With a few simple habits it could have lasted ten minutes.
By the end of this lesson
Note
I can say what to do when production breaks.
Key words
| Word | What it means | Example |
|---|---|---|
| Incident | an unplanned problem that hurts users | |
| Severity | how bad an incident is | SEV1, SEV2, SEV3 |
| On call | the person who answers alerts at that time | |
| Incident lead | the one person who runs the response | |
| Rollback | putting back the last version that worked | |
| Postmortem | a written review after an incident |
The cells students fill in are left blank in this preview.
One lead, a clear picture, a message
The person on call says "I have it" first, so others know.
One person becomes the incident lead. They decide; others do the tasks they are given.
The lead judges the severity: who is hurt, and how badly. Then someone tells support and customers what is known. A short honest update beats silence.
Severity levels in one team
| Level | Example | Response |
|---|---|---|
| SEV1 | Nobody can pay | Everyone needed joins now, day or night |
| SEV2 | Payments are slow for some users | On call works on it now, others join in hours |
| SEV3 | A report page shows old numbers | Fix it in working hours |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think you must find the cause before you act. Not while users are hurt. If the trouble started with a release, roll it back first. The cause can be found calmly afterwards, while customers can pay again.
Stop the harm first
Most incidents start just after a change. So the quickest fix is often a rollback to the version that worked.
One common way: each release sits in its own folder, and a link called current points at the live one. Rolling back moves the link. Try it below.
Roll back a release
mkdir -p shop/releases/v41 shop/releases/v42 && cd shop
printf 'version 41: checkout works\n' > releases/v41/app.txt
printf 'version 42: checkout broken\n' > releases/v42/app.txt
ln -sfn releases/v42 current # the deploy made v42 live
echo "Live now:"
cat current/app.txt
ln -sfn releases/v41 current # roll back: point live at v41 again
echo "After the rollback:"
cat current/app.txt
ls -l current
# Make a releases/v43 folder whose app.txt says "checkout fixed", then make it live.
Learn from it without blame
After the incident, the team writes a postmortem. It is a timeline, the cause, and a few actions to stop it happening again.
It is blameless. It asks how the system let one mistake reach users, not who made it.
People who fear blame hide mistakes. People who do not, report them early.
Part of a postmortem
| Time | What happened |
|---|---|
| 19:02 | Release v42 goes live |
| 19:05 | Alert: payment error rate over 5% for 3 minutes |
| 19:07 | On call takes it and leads; SEV1 declared |
| 19:10 | Status page updated for customers |
| 19:12 | Rollback to v41; errors fall to normal |
| Action | Add a test for card payments in the pipeline |
The cells students fill in are left blank in this preview.
Reliability is a target, not a hope
Teams agree a target for uptime: the share of time the service works.
No target is 100%. Each extra 9 costs much more work, so the team picks what users need.
The table turns a target into minutes of downtime a month.
Worked example: downtime allowed in 30 days
| Uptime target | Down allowed | Minutes in 30 days (43,200) | Roughly |
|---|---|---|---|
| 99% | 1 in 100 | 43,200 divided by 100 = 432 | 7 hours |
| 99.9% | 1 in 1,000 | 43,200 divided by 1,000 = 43 | 43 minutes |
| 99.99% | 1 in 10,000 | 43,200 divided by 10,000 = 4 | 4 minutes |
The cells students fill in are left blank in this preview.