Incident response and forensics lab: investigate a compromised server in your browser

This lab gives each student their own Ubuntu VM holding the logs and files of a web server that was broken into last night: an SSH log with a burst of password guesses, a web server log full of scanners, the accounts file, the crontabs and a list of when files were changed. The student is the incident responder, and has to find out who got in, as whom, when, and what they left behind to come back.

There is no malware and no exploit code, and nothing to run: it is all text, read with the tools every Linux machine has.

What students learn

Reading logs as evidence: searching an SSH log with grep and awk, counting and sorting with sort and uniq, and telling an attack from background noise. The case has distractors on purpose, so a lazy search does not solve it: the address with the most failed logins never gets in, an administrator mistypes a password and then succeeds, and a legitimate new user and an old SSH key sit beside what the intruder left. They build a timeline, and use the time of a change to tell the intruder’s work from the administrator’s.

The task

The student finds six things and writes them in answers.txt, one labelled line each: the address the attack came from, the account that was broken into, the time of the first successful login, the kind of persistence left behind, the file it lives in and the program it keeps running. The README the lab leaves in the home folder says what to find and gives none of the answers.

The evidence is read-only, so the answers have to come from reading it. Every student has the same case.

Run it in your browser

VM labs are set as a lab task in a module, and the sample is on the lab’s page.

  1. Add the lab. In a module you teach, add a task and choose the Ubuntu VM lab card. The page shows a short description of the machine instead of a compose file.
  2. Start from the sample. The page offers ready-made labs to start from. Choose Incident response and forensics. Using a sample replaces the text, and Undo brings yours back.
  3. Publish and start your own copy. Publish, then Start. The lab’s own setup script makes the case the first time the VM starts, in a few seconds. It installs nothing from Ubuntu’s archive unless the image has no Python.
  4. Open the terminal. Open its terminal and read README.md in the home folder: it names the evidence in the case folder and what to find, and gives none of the answers. The answers go in answers.txt, one labelled line each.

A basic lesson on this lab

Incident Response Basics is a small lesson worked in this lab, which is an Ubuntu VM: each step is explained and has a Run button that types its command into the machine’s terminal and shows what it printed. A student lists the evidence, counts the lines in the two logs, reads the first lines of the SSH log and counts the failed passwords.

They look around and leave the lab’s own task, finding the attacker, to the student, so the lesson hands out no answers. A free personal account is given it on the Basics module, which is on Home, every college can copy it from the Library’s Basics folder, and it is deliberately small: it shows a teacher what a lesson worked in a lab is, and the lab does the rest. Copy it into a module of your own to change it.

How it is marked

The lab has a check. It reads answers.txt, which may be written in any capitals and with any spaces, and compares each answer with the case. When one is wrong it names the first wrong field and never says what the answer is. It also refuses a case whose evidence files were changed to suit an answer.

A student has sudo on their own VM, so one who reads root’s files could find the key. That is accepted: the lab is practice, and what is marked is the written answer.

  1. Hand it out. Assign the lab like any task, with a due date if you want one. Each student gets a VM of their own, kept between sessions, holding the same case.
  2. Run the check. On a student’s row in the class list, press Run the check. It starts a copy of their saved machine that is not saved back, runs the lab’s check script as root and shows whether it passed, what it printed and when. The student is not told the result.
  3. Mark it. Read what the check printed: the first line names the first field that is wrong, never the answer. Send the work back with a reason if it needs more.

The definition on GitHub

The file is the whole lab definition, the same text the platform runs: size, setup script that makes the case from a fixed seed, README and check. Copy it and change the generator in the setup to make your own case.

Questions people ask

Is there any malware in the lab?
No. The case is text: logs, an accounts file, crontabs, keys and a list of times. Nothing in it is run, and the addresses are from ranges reserved for documentation or private ranges.
How big a machine does each student need?
One CPU, 1 GB of memory and 10 GB of disk. The lab installs nothing but Python, and only if the image lacks it.
Can a student look up the answer?
A student has sudo on their own VM, so a determined one can read the check’s key in root’s files. The lab is practice, and what your teacher marks is the written answer.