Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Testing detections with known-good and known-bad data

After this lesson: I can test a detection against known-bad and known-good data.

65 min lesson5 commandsDetection engineering lab: Suricata rules, Sigma rules and Zeek-style logs on one private Ubuntu VM

An untested rule is a guess

Testing a detection means running it on data where you already know the answer: traffic that is certainly bad, and traffic that is certainly normal. A good rule fires on all of the first and none of the second.

Teams keep this test data beside their rules and run it on every change, like unit tests. You test rules on made-up captures, in your own lab copy only.

By the end of this lesson

Note

I can test a detection against known-bad and known-good data.

Key words

WordWhat it means
Known-badtest data that holds only attacks
Known-goodtest data that holds only normal activity
Missan attack the rule did not alert on
Test harnessa program that runs every rule on every test case
Beacona program calling home at a steady rhythm
Gapthe time between one connection and the next

The cells students fill in are left blank in this preview.

The one rule

Important

Test data here is made up. Never fill a test set by attacking a real machine, and never copy real people's traffic into one without permission. A lab copy of your own is the place to make bad traffic.

Two test sets catch two kinds of mistake

Known-bad data finds misses: an attack the rule should have caught. Known-good data finds false positives: normal activity it should have left alone.

The lab's det test is a small test harness. It runs every rule in a file on both captures and counts the alerts. known-bad.pcap holds five bad events: two updater lookups and three check-ins.

Move into the lab's folder

Command
cd ~/detect

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Test the example rules

Command
det test rules/examples.rules

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: what the harness says

RuleKnown-bad alertsKnown-good alertsVerdict
1000001 first try2 of 2 lookups3Catches all, too noisy
1000002 tuned lookup2 of 2 lookups0Passes
1000003 check-in3 of 3 check-ins0Passes, one alert each
1000004 with a limit1 for one machine0Passes, one alert per machine

The cells students fill in are left blank in this preview.

Try it: a harness in eight lines

Code: python
def detect(name):
    return name.endswith("cdn-update.example")

cases = [("a1.cdn-update.example", True), ("cdn-updates.example", False),
         ("mycdn-update.example", False), ("cdn-update.example", True)]
for name, bad in cases:
    got = detect(name)
    print("ok  " if got == bad else "FAIL", name)
# Put a dot in front of the name and the ending. Which line changes?

A common mistake

Warning

Many people test a rule only on the attack it was written for. It catches it, so they ship it. A rule that alerts on everything also catches it. Only known-good data shows the noise, so every rule needs both sets.

Some attacks show as a rhythm

Not every detection matches a name. Updater calls home about every 60 seconds, a beacon. People and browsers are not that regular.

The first Run makes the logs again and the second shows updater's first three connections with their times. The third lists the gaps between them, rounded, and counts each.

Make the logs from the capture

Command
zeek-lite -r pcaps/office.pcap

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Updater's first three connections

Command
zeek-cut ts id.orig_h id.resp_h < conn.log | grep 198.51.100.23 | head -n 3

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

The gaps between them, counted

Command
zeek-cut ts id.resp_h < conn.log | awk '$2 == "198.51.100.23" { if (p) print int($1 - p + 0.5); p = $1 }' | sort -n | uniq -c

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

About this lab and its source: Detection engineering lab: Suricata rules, Sigma rules and Zeek-style logs on one private Ubuntu VM

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.