Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Tuning detections with precision, recall and ATT&CK

After this lesson: I can measure a detection's precision and recall and map it to ATT&CK.

65 min lesson4 commandsDetection engineering lab: Suricata rules, Sigma rules and Zeek-style logs on one private Ubuntu VM

Five hundred alerts and one attack

Tuning a detection means measuring how often it is right and how much it misses, then changing it until both are good enough. Precision and recall are the two numbers teams use.

An analyst who sees five hundred alerts a day stops reading them, and the one real attack goes by. You measure made-up rules on a made-up office, in your own lab copy only.

By the end of this lesson

Note

I can measure a detection's precision and recall and map it to ATT&CK.

Key words

WordWhat it meansExample
Precisionthe share of alerts that were real attacks
Recallthe share of attacks that raised an alert
Alert fatiguetiredness that makes people stop reading alerts
ATT&CKa public catalogue of attacker techniques, each with an idT1059.001
Coveragewhich techniques at least one rule detects

The cells students fill in are left blank in this preview.

The one rule

Important

Measure rules on data you are allowed to use: your lab copy, or a network whose owner has agreed in writing. ATT&CK describes what attackers do so defenders can find it, never as a guide to doing it.

Precision and recall answer different questions

Precision asks: when the rule alerted, how often was it right? It is the true hits divided by all alerts. Recall asks: of all the attacks, how many did it catch? It is the true hits divided by the attacks.

The office capture holds 5 updater lookups and 20 check-ins. The Run counts each rule's alerts.

Move into the lab's folder

Command
cd ~/detect

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Count each rule's alerts

Command
det count pcaps/office.pcap rules/examples.rules

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: scoring the lookup rules

RuleAlertsTrue hitsAttacksPrecisionRecall
1000001 first try8555 / 8 = 0.6255 / 5 = 1.0
1000002 tuned5555 / 5 = 1.05 / 5 = 1.0

The cells students fill in are left blank in this preview.

Try it: work out the two numbers

Code: python
alerts = 8      # rule 1000001 on the office capture
true_hits = 5   # alerts on real updater lookups
attacks = 5     # updater lookups in the capture

precision = true_hits / alerts
recall = true_hits / attacks
print("precision", round(precision, 3), "recall", round(recall, 3))
# Rule 1000002 gave 5 alerts, all real. Change the numbers for it.

A common mistake

Warning

Many people think a rule should catch everything whatever it costs. A rule with perfect recall and poor precision floods the team, and alert fatigue then hides real attacks. Aim for both, and count the alerts a day as well.

Count the alerts, not only the attacks

Rules 1000003 and 1000004 both catch the infected machine, so both have a recall of one machine out of one. The first sends 20 alerts in twenty minutes, about 1,400 a day. The second sends one an hour.

A team sets a budget of alerts a day for each rule, and a rule over it is tuned or merged.

ATT&CK shows what your rules cover

MITRE ATT&CK is a public catalogue of attacker techniques. T1071.001 is calling home over the web, T1071.004 over DNS, T1059.001 running PowerShell and T1033 asking who you are.

Tagging every rule with its technique turns a folder of rules into a coverage map: what you detect, and what you do not. The Runs read the tags.

The techniques in the Suricata rules

Command
grep -o "mitre_technique_id [A-Z0-9.]*" rules/examples.rules | sort | uniq -c

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

The techniques in the Sigma rules

Command
grep -h "attack\.t" sigma/*.yml

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

About this lab and its source: Detection engineering lab: Suricata rules, Sigma rules and Zeek-style logs on one private Ubuntu VM

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.