Observability lab: Prometheus and Grafana, with an alerting rule for the student to write

This lab gives each student a small web server to watch and the tools that watch it: an exporter that turns the server’s status page into metrics, Prometheus collecting them every five seconds, and Grafana, already signed in and already connected to Prometheus. The student asks questions in PromQL, draws a panel, then writes an alerting rule, breaks the server on purpose and watches the alert go from pending to firing.

It is designed by us from four public images, so there is nothing vulnerable to attack: only a system to watch. It is not an attack lab. The lab copy has no route to the internet and none to any other student.

What students learn

What monitoring is for and how its pieces fit: a program that publishes numbers, an exporter for software that does not, a collector that scrapes them on a timer and stores them, and a dashboard that draws them. They ask questions in PromQL, such as nginx_up and the rate of requests over a minute, both in the Prometheus page and with curl against its API, and make a Grafana panel from the same query. The centre of the lab is an alerting rule: they write it into a rules file, reload Prometheus, see it listed as inactive, break the server and watch the alert go pending, then firing, and clear when they put the server right. It suits DevOps, operations, cloud and systems units; check your own awarding body’s unit for the exact wording.

Run it in your browser

The web server is the service called app, its exporter is exporter, and the lab’s terminal reaches Prometheus as prometheus and Grafana as grafana. A docker command in the terminal can look at, enter, stop, start and restart the lab’s services and read their logs: it cannot create, pull or remove anything. Prometheus’s configuration and rules, and the student’s Grafana dashboards, are on named volumes, so they are still there after the lab is stopped and started again. The measurements are not saved, so a restarted lab starts with an empty graph.

  1. Open the Container page. Open Build on the bar and choose Run a container. Anybody may open it, and the Git lab and the SQL lab start with no account. Every other lab needs a free account: create one, sign in, then press Start.
  2. Find the lab. The Sample labs card lists every sample. Find Observability: Prometheus and Grafana.
  3. Press Start. Start runs it as it is, in one press. Edit the file first opens the compose file so you can change it before you start.
  4. Open it. On the lab’s page the lab bar has two links, Prometheus and Grafana, already signed in and connected to Prometheus. In Prometheus, run nginx_up. In the terminal beside the lab, curl http://prometheus:9090/api/v1/query?query=nginx_up asks the same question. To write a rule, run docker exec -it prometheus sh and put a file ending in .yml in /etc/prometheus/rules, reload with curl -X POST http://prometheus:9090/-/reload, then docker stop app and watch the Alerts page. docker start app puts the server right.

What to ask students to hand in

The rule itself, a screenshot of it firing on the Prometheus Alerts page, and a Grafana panel showing the same signal. For the understanding, a sentence or two on what the rule asks, why it waits ten seconds before it fires, and what a person on call would do when they saw it. For stronger work ask for a second rule on a different signal, such as the request rate.

Nothing is marked automatically. The lab’s limits are worth saying out loud: the alert can be watched but nothing is notified, because there is no email or chat for it to reach.

A basic lesson on this lab

Observability Basics is a small lesson worked in this lab, with each step explained and a Run button that types its command into the student’s terminal. It asks Prometheus a question, writes one alerting rule from the terminal, stops the web server and shows the alert go pending and then firing, and starts it again. Drawing a panel is left to the student.

A free personal account is given it on the Basics module, which is on Home, every college can copy it from the Library’s Basics folder, and a new college opens with it in a module called Lab Basics (Sample). It is deliberately small: it shows a teacher what a lesson worked in a lab is, and the lab does the rest. Copy it into a module of your own to change it.

The code on GitHub

The compose file runs four public images, each pinned to a version and run unmodified: nginx (BSD-2-Clause), the NGINX Prometheus exporter and Prometheus (both Apache-2.0) and Grafana (AGPL-3.0). None is distributed by us. The configuration each needs is written by its own entrypoint, and Prometheus’s is written only on the first start, so what a student changes is not put back. The repository’s test plays a student through the lab, including a rule that fires and a stop and start, under the same limits the platform applies. Grafana’s plugin catalogue and update checks are switched off, because the lab has no way out to the internet.

Set it as work

Say in the task what rule to write and what to hand in.

  1. Make the lab. In your module add a task and choose the Container lab card, then In a GitHub repository. Give https://github.com/mitsurutech/lab-observability with the branch main. The compose file is the default, compose.yaml. Fork the repository first if you want to change it.
  2. Publish and try it. Publish pins the commit you were shown, so a later change in the repository does not alter a lab you have already set. Start your own copy first.
  3. Hand it out. Assign it like any task, with a due date if you want one, and say in the task what to do and what to hand in.
  4. Mark what is handed in. Open a student’s copy and the work they handed in, mark it, and send it back with a reason if it needs more.

Questions people ask

Can an alert send an email or a message?
No. The lab has no way out to the internet and no mail service in it, so an alert can be seen on the Alerts page and in the API but is not sent anywhere. That is worth saying to a class: the rule is the part a student writes, and where it is sent is a separate piece, Alertmanager, that this lab does not include.
What if a student breaks the Prometheus configuration?
The file is written once, on the first start, and is theirs afterwards, so a restart does not put it back. When a reload fails, for example because of a mistake in a rule file, the reload command prints the reason and Prometheus keeps the rules it had before. To start from the original, remove the file .lab-seeded in /etc/prometheus and restart the service, or begin a fresh copy.
Does the observability lab mark the alerting rule automatically?
No. Nothing is marked automatically, so ask students to hand in the rule, a screenshot of it firing on the Prometheus Alerts page and a Grafana panel showing the same signal. A sentence on why the rule waits ten seconds before it fires shows whether they understood it.