Five hundred alerts and one attack
Tuning a detection means measuring how often it is right and how much it misses, then changing it until both are good enough. Precision and recall are the two numbers teams use.
An analyst who sees five hundred alerts a day stops reading them, and the one real attack goes by. You measure made-up rules on a made-up office, in your own lab copy only.
By the end of this lesson
Note
I can measure a detection's precision and recall and map it to ATT&CK.
Key words
| Word | What it means | Example |
|---|---|---|
| Precision | the share of alerts that were real attacks | |
| Recall | the share of attacks that raised an alert | |
| Alert fatigue | tiredness that makes people stop reading alerts | |
| ATT&CK | a public catalogue of attacker techniques, each with an id | T1059.001 |
| Coverage | which techniques at least one rule detects |
The cells students fill in are left blank in this preview.
The one rule
Important
Measure rules on data you are allowed to use: your lab copy, or a network whose owner has agreed in writing. ATT&CK describes what attackers do so defenders can find it, never as a guide to doing it.
Precision and recall answer different questions
Precision asks: when the rule alerted, how often was it right? It is the true hits divided by all alerts. Recall asks: of all the attacks, how many did it catch? It is the true hits divided by the attacks.
The office capture holds 5 updater lookups and 20 check-ins. The Run counts each rule's alerts.
Move into the lab's folder
cd ~/detectShown here, not run. Press Start lab to type it into your own lab’s terminal.
Count each rule's alerts
det count pcaps/office.pcap rules/examples.rulesShown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: scoring the lookup rules
| Rule | Alerts | True hits | Attacks | Precision | Recall |
|---|---|---|---|---|---|
| 1000001 first try | 8 | 5 | 5 | 5 / 8 = 0.625 | 5 / 5 = 1.0 |
| 1000002 tuned | 5 | 5 | 5 | 5 / 5 = 1.0 | 5 / 5 = 1.0 |
The cells students fill in are left blank in this preview.
Try it: work out the two numbers
alerts = 8 # rule 1000001 on the office capture
true_hits = 5 # alerts on real updater lookups
attacks = 5 # updater lookups in the capture
precision = true_hits / alerts
recall = true_hits / attacks
print("precision", round(precision, 3), "recall", round(recall, 3))
# Rule 1000002 gave 5 alerts, all real. Change the numbers for it.
A common mistake
Warning
Many people think a rule should catch everything whatever it costs. A rule with perfect recall and poor precision floods the team, and alert fatigue then hides real attacks. Aim for both, and count the alerts a day as well.
Count the alerts, not only the attacks
Rules 1000003 and 1000004 both catch the infected machine, so both have a recall of one machine out of one. The first sends 20 alerts in twenty minutes, about 1,400 a day. The second sends one an hour.
A team sets a budget of alerts a day for each rule, and a rule over it is tuned or merged.
ATT&CK shows what your rules cover
MITRE ATT&CK is a public catalogue of attacker techniques. T1071.001 is calling home over the web, T1071.004 over DNS, T1059.001 running PowerShell and T1033 asking who you are.
Tagging every rule with its technique turns a folder of rules into a coverage map: what you detect, and what you do not. The Runs read the tags.
The techniques in the Suricata rules
grep -o "mitre_technique_id [A-Z0-9.]*" rules/examples.rules | sort | uniq -cShown here, not run. Press Start lab to type it into your own lab’s terminal.
The techniques in the Sigma rules
grep -h "attack\.t" sigma/*.ymlShown here, not run. Press Start lab to type it into your own lab’s terminal.