The shop server acted strangely overnight
On Wednesday morning, the owner of an online phone-case shop notices the server is slow. There is a program running that nobody recognises.
She does not want to panic, and she does not want to guess. She wants to know what happened, when it happened and how to stop it happening again.
The answers are in the logs, if somebody reads them carefully.
The one rule
Warning
This lab is a private machine holding copies of the evidence from a pretend break-in. Work only on your own copy, with your teacher's permission. There is no malware and nothing to run, only text to read. Never look through logs or files on a system you are not allowed to.
By the end of this lesson
Note
I can report what happened in an attack, using its logs.
Key words
| Word | What it means |
|---|---|
| Incident | an event that harms or threatens a system |
| Log | a file where a program writes down what happens |
| Evidence | the files that show what happened |
| Timeline | the events of an incident in time order |
| Persistence | something an intruder leaves behind to get back in |
| Containment | stopping an incident from spreading |
The cells students fill in are left blank in this preview.
Responding follows the same steps every time
A good response does not start by deleting things. It follows steps.
Detect that something is wrong. Contain it, for example by taking the server off the network. Investigate from the evidence. Remove what the intruder left. Recover the service. Learn, and write a report.
Investigating needs evidence that nobody has changed. So the first rule is: copy it, and work on the copy.
What logs were copied for you?
ls case/var/logShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Try to add a file to the evidence
touch case/note.txtShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Counting shows what is normal noise
auth.log records every attempt to sign in over the network. grep -c counts the lines that contain some text.
Any server on the internet gets thousands of guesses from robots. Most are noise. The useful question is what is unusual: a guess that worked, or one address that behaves differently.
Run the three counts. Then think about what they do and do not prove.
Guesses at accounts that do not exist
grep -c "Invalid user" case/var/log/auth.logShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Every wrong password
grep -c "Failed password" case/var/log/auth.logShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Every sign-in that worked
grep -c "Accepted" case/var/log/auth.logShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Worked example: what the counts tell you
| Count | It printed | What it tells you | What it does not prove |
|---|---|---|---|
| Invalid user | 297 | Many guesses at made-up names | Who was behind them |
| Failed password | 333 | 333 minus 297 is 36 wrong passwords for real accounts | That any guess worked |
| Accepted | 6 | Six sign-ins worked | Which were the owner and which an intruder |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think the address with the most failed guesses must be the attacker. Noisy robots fail all day and never get in. Look for the address that failed and then succeeded, because that is the one that got in.
A timeline puts every clue in order
Clues make sense in time order. mtimes.txt lists when files were last changed: date, time, owner and path.
Order helps you rule things out. A change made weeks before the intruder arrived cannot be theirs. A file changed minutes after their first sign-in deserves a close look.
This Run counts how many files changed on each date. It shows the shape of the story, not the answer.
How many changes on each date?
cut -d " " -f 1 case/mtimes.txt | sort | uniq -cShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A report answers five questions
Write the report for the shop owner, not for an expert. Answer five questions in plain words.
What happened? When, from the first sign to the last? How did they get in? What did they leave or take? What should change, so it cannot happen again?
Give each fact with the log line that proves it. A claim without evidence is a guess.
Going further: the busiest hour
Use failures_by_hour from your assignment on a log of your own with ten lines over three hours. Which hour was busiest?
Then write a short report for a made-up shop: one sentence for each of the five questions, each with the log line that proves it.