Four hundred alerts before lunch
Alert triage is the job of deciding which security alerts to look at first. A busy team can get hundreds of alerts in a morning, and most are false alarms.
If you treat them all alike, the one real attack waits in the queue. You will learn to rank alerts by what matters most.
By the end of this lesson
Note
I can rank a queue of alerts by how much each one matters.
Key words
| Word | What it means | Example |
|---|---|---|
| Alert | a message that something may be wrong | |
| Triage | sorting alerts so the most important come first | |
| Severity | how bad the alert says the problem is | low, medium, high, critical |
| Asset | a machine or account the business needs | |
| False alarm | an alert for something that is not a problem | |
| Escalate | pass an alert to someone with more power to act |
The cells students fill in are left blank in this preview.
Priority comes from severity and from the asset
An alert has a severity, but severity is only half the story. A critical alert on a test machine nobody uses matters less than a medium alert on the server that takes customers' orders.
So a team scores two things: how bad the alert is, and how much the asset matters. Multiply them. The biggest score goes first.
Worked example: scoring four alerts
| Alert | Severity points | Asset points | Score | Place |
|---|---|---|---|---|
| Strange login | 3 (high) | 3 (customer orders) | 9 | First |
| Port scan | 2 (medium) | 3 (customer orders) | 6 | Second |
| Virus found | 4 (critical) | 1 (test machine) | 4 | Third |
| Certificate old | 1 (low) | 2 (office) | 2 | Fourth |
The cells students fill in are left blank in this preview.
Count a queue by severity
printf 'low\nhigh\nlow\ncritical\nhigh\nlow\n' | sort | uniq -c | sort -rnShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A rule that cries wolf stops being read
A false alarm is an alert for something that is not a problem. A few are normal. Too many teach people to ignore alerts, and then a real one is missed.
Measure it. Count the alerts that turned out false and divide by all the alerts. A rule that is false 95 times in 100 needs to be fixed or removed.
Work out a false alarm rate
awk 'BEGIN{print 38 * 100 / 40}'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Count the web requests answered not found
grep -c " 404 " case/var/log/nginx/access.logShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people think every critical alert must be handled first. Severity is the rule's guess in general terms. A critical alert on a machine that holds nothing may wait. A medium alert on your main server may not. Always weigh the asset too.
Worked example: false alarm rates
| Rule | Alerts | False | False alarm rate |
|---|---|---|---|
| Any failed sign-in | 400 | 392 | 98 out of 100 |
| Ten failures in one minute | 20 | 8 | 40 out of 100 |
| Login from a new country | 5 | 1 | 20 out of 100 |
The cells students fill in are left blank in this preview.
Three questions turn an alert into a decision
For each alert at the top of the queue, ask three things. Is it real? How far did it go? Who must know?
If it is real and has touched something important, escalate: pass it on at once to the person who can act. Write down what you checked, even when the answer is a false alarm.
Going further
Your team gets 30 alerts a day. A new rule would add 60 more, and you expect 57 to be false.
Write two sentences. Should the team switch it on? What would you change in the rule first?