An untested rule is a guess
Testing a detection means running it on data where you already know the answer: traffic that is certainly bad, and traffic that is certainly normal. A good rule fires on all of the first and none of the second.
Teams keep this test data beside their rules and run it on every change, like unit tests. You test rules on made-up captures, in your own lab copy only.
By the end of this lesson
Note
I can test a detection against known-bad and known-good data.
Key words
| Word | What it means |
|---|---|
| Known-bad | test data that holds only attacks |
| Known-good | test data that holds only normal activity |
| Miss | an attack the rule did not alert on |
| Test harness | a program that runs every rule on every test case |
| Beacon | a program calling home at a steady rhythm |
| Gap | the time between one connection and the next |
The cells students fill in are left blank in this preview.
The one rule
Important
Test data here is made up. Never fill a test set by attacking a real machine, and never copy real people's traffic into one without permission. A lab copy of your own is the place to make bad traffic.
Two test sets catch two kinds of mistake
Known-bad data finds misses: an attack the rule should have caught. Known-good data finds false positives: normal activity it should have left alone.
The lab's det test is a small test harness. It runs every rule in a file on both captures and counts the alerts. known-bad.pcap holds five bad events: two updater lookups and three check-ins.
Move into the lab's folder
cd ~/detectShown here, not run. Press Start lab to type it into your own lab’s terminal.
Test the example rules
det test rules/examples.rulesShown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: what the harness says
| Rule | Known-bad alerts | Known-good alerts | Verdict |
|---|---|---|---|
| 1000001 first try | 2 of 2 lookups | 3 | Catches all, too noisy |
| 1000002 tuned lookup | 2 of 2 lookups | 0 | Passes |
| 1000003 check-in | 3 of 3 check-ins | 0 | Passes, one alert each |
| 1000004 with a limit | 1 for one machine | 0 | Passes, one alert per machine |
The cells students fill in are left blank in this preview.
Try it: a harness in eight lines
def detect(name):
return name.endswith("cdn-update.example")
cases = [("a1.cdn-update.example", True), ("cdn-updates.example", False),
("mycdn-update.example", False), ("cdn-update.example", True)]
for name, bad in cases:
got = detect(name)
print("ok " if got == bad else "FAIL", name)
# Put a dot in front of the name and the ending. Which line changes?
A common mistake
Warning
Many people test a rule only on the attack it was written for. It catches it, so they ship it. A rule that alerts on everything also catches it. Only known-good data shows the noise, so every rule needs both sets.
Some attacks show as a rhythm
Not every detection matches a name. Updater calls home about every 60 seconds, a beacon. People and browsers are not that regular.
The first Run makes the logs again and the second shows updater's first three connections with their times. The third lists the gaps between them, rounded, and counts each.
Make the logs from the capture
zeek-lite -r pcaps/office.pcapShown here, not run. Press Start lab to type it into your own lab’s terminal.
Updater's first three connections
zeek-cut ts id.orig_h id.resp_h < conn.log | grep 198.51.100.23 | head -n 3Shown here, not run. Press Start lab to type it into your own lab’s terminal.
The gaps between them, counted
zeek-cut ts id.resp_h < conn.log | awk '$2 == "198.51.100.23" { if (p) print int($1 - p + 0.5); p = $1 }' | sort -n | uniq -cShown here, not run. Press Start lab to type it into your own lab’s terminal.