The customers knew first
On Monday the shop's checkout failed for one customer in ten. The team found out an hour later, from angry posts online.
The problem had shown up in the shop's own numbers within a minute. Nobody was looking at them.
This lesson is about seeing trouble before your customers tell you.
By the end of this lesson
Note
I can use logs, metrics and alerts to see a system's health.
Key words
| Word | What it means | Example |
|---|---|---|
| Log | a line a program writes about one thing that happened | |
| Metric | a number measured again and again over time | |
| Error rate | the percentage of requests that failed | |
| Latency | how long a request takes | 120ms |
| Alert | a message that calls a person to act | |
| Dashboard | a screen of a system's main metrics |
The cells students fill in are left blank in this preview.
Logs say what happened
A program writes a log line when something happens. Each line has a time, a level and a message.
The level says how serious it is: INFO is normal, WARN is odd, ERROR is a failure.
Logs answer "what went wrong with order 1003?". Search them with grep.
Search a log
mkdir -p logs && cd logs
cat > app.log <<'EOF'
2026-10-21 09:00:01 INFO order 1001 paid in 120ms
2026-10-21 09:00:04 INFO order 1002 paid in 135ms
2026-10-21 09:00:09 ERROR order 1003 payment timed out after 5000ms
2026-10-21 09:00:12 INFO order 1004 paid in 128ms
2026-10-21 09:00:15 ERROR order 1005 payment timed out after 5000ms
2026-10-21 09:00:19 WARN order 1006 paid in 2400ms
EOF
echo "Lines in the log:"
wc -l < app.log
echo "The errors:"
grep ERROR app.log
echo "How many errors:"
grep -c ERROR app.log
# Change the last grep so it counts the WARN lines instead.
Metrics are numbers over time
Reading every log line does not scale. A metric turns them into one number a minute.
Three metrics tell you most of a web service's health: how many requests, the error rate and the latency.
A change in the number is the warning. One failure in 200 is normal; 21 in 210 is not.
Worked example: the error rate
| Minute | Requests | Failed | Working | Error rate |
|---|---|---|---|---|
| 09:00 | 200 | 2 | 2 x 100, divided by 200 | 1% |
| 09:01 | 180 | 9 | 9 x 100, divided by 180 | 5% |
| 09:02 | 210 | 21 | 21 x 100, divided by 210 | 10% |
| 09:03 | 190 | 19 | 19 x 100, divided by 190 | 10% |
The cells students fill in are left blank in this preview.
Work out the error rate
mkdir -p logs && cd logs
# One line a minute: the time, requests, failed requests
printf '09:00 200 2\n09:01 180 9\n09:02 210 21\n09:03 190 19\n' > metrics.txt
awk '{ printf "%s requests %d error rate %d%%\n", $1, $2, $3 * 100 / $2 }' metrics.txt
# Add a line for 09:04 with 200 requests and 40 failed, then run again.
A common mistake
Warning
Many people think more alerts make a system safer. They do the opposite. When alerts fire for things nobody needs to fix, people learn to ignore them. Then they miss the real one, so alert only when users are hurt and a person must act.
An alert calls a person to act
Nobody watches a dashboard all night. An alert watches a metric for you.
A good alert has a threshold and a time: "error rate over 5% for 3 minutes in a row". One bad minute is often noise.
It goes to the person on call, with a link to the dashboard and the logs.