How to find an attack in a log
Finding an attack in a log means reading a server's record of sign-ins and picking out the address that broke in. It matters because one break-in hides among hundreds of harmless failures, and it must be contained before it spreads.
Most failures are bots that never get in. A few are the owner mistyping a password. One address is different: it fails again and again, and then it succeeds.
By the end of this lesson
Note
I can find a break-in in a log, contain it and name the fix.
Key words
| Word | What it means |
|---|---|
| Log | a file where a system writes down what happened, one line each |
| Brute force | trying many passwords until one works |
| Source address | the network address a connection came from |
| Containment | stopping the harm from spreading before you fix the cause |
| Noise | log lines that look alarming and mean nothing |
| Pivot | the single fact that separates an attack from noise |
The cells students fill in are left blank in this preview.
Count first, then ask who got in
The case is a copy of a server's SSH log, in the folder case. It is read-only, so the evidence stays as it was.
A big count is not an attack on its own. First count the failures for each address. Then ask which addresses ever got in.
Count failed sign-ins for each address
grep "Failed password" case/var/log/auth.log | awk '{for (i = 1; i <= NF; i++) if ($i == "from") print $(i + 1)}' | sort | uniq -c | sort -rn | head -n 3Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
The top three
The Run printed three lines, each a count and an address. On the machine this was tried on they were 260, 70 and 3. The addresses are the same for every student in this lab.
The address with 260 failures is the loudest. Loud does not mean dangerous, so do not stop here.
Who was let in, and from where?
awk '$6 == "Accepted" {print $9, $11}' case/var/log/auth.log | sort | uniq -cShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Save the addresses that got in
awk '$6 == "Accepted" {print $11}' case/var/log/auth.log | sort -u > accepted.txtShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Which loud address also got in?
awk '$6 == "Failed" {for (i = 1; i <= NF; i++) if ($i == "from") print $(i + 1)}' case/var/log/auth.log | sort | uniq -c | awk '$1 >= 20 {print $2}' | sort | comm -12 - accepted.txt | tee suspect.txtShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Show its first successful sign-in
grep Accepted case/var/log/auth.log | grep -F -f suspect.txt | head -n 1Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the four Runs found
The first printed three groups. The administrator signed in from the office address, and once from the address where she had mistyped. A third account was let in twice from the suspect's address.
The pivot Run printed one address. It failed at least twenty times and then succeeded. The last Run showed the line of its first success, with the time and the account.
Contain first, then fix the cause
Once you know, do the steps in this order. Containment is quick and can be undone. The fix takes longer and must be right.
Do not delete anything yet. The evidence is how you learn what else was touched.
Worked example: from log line to fix
| Step | What you do | Why |
|---|---|---|
| Detect | Count failures by address, then see who got in | Finds the pivot |
| Contain | Lock the account and refuse the address | Stops the intruder using it again |
| Fix the cause | SSH keys only, a rule for the port, no password sign-in | The same guess cannot work again |
| Check | Look for what was left behind: keys, jobs, new accounts | Closes a second way back in |
The cells students fill in are left blank in this preview.
Practise containment on an account of your own
You will not touch any account in the case. Make a throwaway account called demo7, lock it, read its status and remove it.
A locked account cannot sign in with a password, and nothing is deleted. That is why it is the first step.
Make the throwaway account
sudo useradd -m -s /bin/bash demo7Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Lock its password
sudo passwd -l demo7Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Read its status
sudo passwd -S demo7Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Remove the account and its folder
sudo userdel -r demo7Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the account Runs printed
The lock Run printed passwd: password changed.. The status Run printed a line that begins demo7 L. The letter L means locked. An account that can sign in shows P.
The remove Run may print a warning about a mail spool. That is harmless, because the account never had mail.
Delete the two files you made
rm accepted.txt suspect.txtShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A common mistake
Warning
Many people block the address with the most failures and call the incident closed. That address never got in. The one that did is quieter, and it is still inside. Always ask who succeeded, then check what they changed.
Going further: what else did the address do?
Look at the web server's log, case/var/log/nginx/access.log. Run the two saving Runs again to remake suspect.txt, then count the requests from that address with grep -c -F -f suspect.txt. Delete both files afterwards.
Write one sentence: what was it asking for, an hour before it began guessing passwords?