Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Detection Engineering Basics

The whole lesson is free and interactive. Read it here, or press Start lab to run it for real in a live lab, with no account needed.

Virtual machine lab9 min read6 commands

Every alert started as a rule

A security team gets an alert: a laptop looked up a name that a known piece of malware uses. Somebody wrote the rule that saw it, and tested it so it does not fire on every normal lookup. That work is detection engineering. Use these tools only on your own lab copy, and on a real network only with written permission.

By the end of this lesson

Note

I can run a detection rule on network traffic and read its alerts.

Key words

  • Packet capture: a file of the packets that crossed a network.
  • Zeek: a tool that turns traffic into logs.
  • Suricata: a tool that checks traffic against rules and raises alerts.
  • Sigma: a format for writing rules about logs.
  • False positive: an alert on something harmless.

Starter: what you already know

Note

Answer in your head. 1. What does DNS do? 2. What is a log file? 3. Is an alert always an attack? You need the third answer in a moment.

Before you start

Note

Press Start the lab on the bar at the top of this lesson. This lab is a whole Ubuntu machine of your own, so it takes a little longer to come up than a container. Each Run button types its command into the machine's terminal, one after another in the same session. If one says the machine is still starting, wait a few seconds and press it again. To watch them, or to type your own, press Open terminal on the bar.

A capture becomes logs

Your files are in the folder detect. The first command moves into it. pcaps/office.pcap is twenty minutes of a made-up office's traffic. zeek-lite turns it into the logs Zeek writes: conn.log, dns.log and http.log. Zeek is not in Ubuntu's archive, so zeek-lite is the lab's small model of it, with the same format. The third command counts every name the office looked up.

Command
cd detect

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Command
zeek-lite -r pcaps/office.pcap

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Command
zeek-cut query answers < dns.log | sort | uniq -c

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Check

Tip

Which names look like a tool calling home? Answer: the five under cdn-update.example, all answered 198.51.100.23. Two other names, cdn-updates.example and cdn-update.example.org, only look like them.

A rule raises alerts

rules/examples.rules holds four Suricata rules for a made-up tool called updater. det is the lab's helper: det count runs Suricata on a capture with those rules only and counts each rule's alerts. The first try fires 8 times, the tuned rule 5. The check-in rule fires 20 times, and the same rule with a limit only once.

Command
det count pcaps/office.pcap rules/examples.rules

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

A common mistake

Warning

Many people think a rule that fires more is a better rule. It is not. The first try fired 8 times because 3 lookups were of names that only look like updater's. Every false positive costs somebody time, so a rule is tested before it is trusted.

Test on known-bad and known-good data

det test runs the rules on two captures whose answers are known. known-bad.pcap holds five bad events and known-good.pcap only normal traffic. A good rule fires on all of the first and none of the second. Only the first try fires on known-good, 3 times.

Command
det test rules/examples.rules

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Check

Tip

Which rule needs tuning, and why? Answer: rule 1000001, because it alerted 3 times on known-good traffic. Those are false positives.

A Sigma rule searches logs

sigma/whoami.yml is a Sigma rule for Windows events: it matches any run of whoami. det sigma converts it into SQL with the real Sigma tools and runs it on the lab's events. It finds two: a user on PC-15 straight after a suspicious PowerShell, and an admin on PC-13 doing their job.

Command
det sigma sigma/whoami.yml

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Exit check

Tip

Answer without looking back. 1. What does Zeek turn traffic into? 2. What does Suricata do with a rule? 3. What is a false positive? 4. Why test on known-good data? Answers: logs; checks traffic against it and raises alerts; an alert on something harmless; to see the false positives. Now open README.md in your home folder with cat ~/README.md: you write rules for a made-up tool called quietbird.

For teachers

Tip

This lesson is a small example of a lesson worked in a lab: the steps and the Run buttons are the lesson, and the lab is each student's own copy. This lesson is worked in a VM lab: each student has an Ubuntu machine of their own with Suricata, the Sigma tools and the lab's model of Zeek, and a Run types into its terminal and shows what it printed. Key learning points: 1. Zeek-style logs summarise traffic. 2. A Suricata rule raises alerts on matching traffic. 3. A rule is tested on known-bad and known-good data. 4. Sigma rules search logs. Common mistake: thinking more alerts means a better rule. Equipment: the lab and a browser, and a free or college account, since the setup installs from PyPI. Tip: ask the class to guess which names are look-alikes before the count. The lab itself has a check that marks two files with traffic and logs it makes afresh. Copy it into your own module and change it, or build one from scratch with the Lab menu in the lesson builder.

About this lab and its source: Detection engineering lab: Suricata rules, Sigma rules and Zeek-style logs on one private Ubuntu VM

Teaching a class? Set these lessons for your students with marking and a lab for every student.