What you will do
Running software is only half of looking after it: the other half is knowing when it goes wrong before somebody tells you. Monitoring collects numbers about a program over time, and an alert is a rule that watches those numbers and speaks up. Your lab has a web server, a collector called Prometheus and a drawing tool called Grafana. You will ask a question, write one alerting rule, break the server on purpose and watch the alert fire.
Before you start
Note
Press Start the lab on the bar at the top of this lesson and wait until it says it is running. The four containers are app, exporter, prometheus and grafana, and they find each other by those names. Each Run button below types its command into your lab's terminal, one after another in the same session. To watch them, or to type your own, press Open terminal on the bar.
1. Ask Prometheus a question
Prometheus keeps every number it has collected. In its language, PromQL, the simplest question is the name of one. nginx_up is 1 while the web server answers and 0 when it does not. The answer is JSON: look for "value":[ followed by a time and "1". The same question typed into the Prometheus page in your browser draws the answer as a table or a graph. -w "\n" ends the answer with a new line, so it sits on its own line under the command.
curl -s -w "\n" 'http://prometheus:9090/api/v1/query?query=nginx_up'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
2. Write an alerting rule
A rule says "when this question is true for ten seconds, raise an alert". This one asks nginx_up == 0, in other words the server is down. The first command writes it, on one line, into the rules folder on the Prometheus container, where Prometheus reads every .yml file. It prints nothing. Then the second command asks Prometheus to read its files again, and prints nothing either: on the Prometheus page, Alerts now lists AppDown, green because nothing is wrong.
docker exec prometheus sh -c 'echo "$0" > /etc/prometheus/rules/lab.yml' '{"groups":[{"name":"lab","rules":[{"alert":"AppDown","expr":"nginx_up == 0","for":"10s","labels":{"severity":"page"}}]}]}'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
curl -s -X POST http://prometheus:9090/-/reloadShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
3. Break the server and watch it fire
docker stop app stops the web server, and prints its name. Prometheus notices within about ten seconds, and the alert is "state":"pending" while it waits out the ten seconds the rule asked for. Run the second command twice, a few seconds apart: the answer changes to "state":"firing". Then docker start app puts the server right: press the second Run once more after fifteen seconds and the answer is "alerts":[] again. To finish, open Grafana, press Explore, choose the Prometheus data source and run nginx_up: that is the line a dashboard panel draws.
docker stop appShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
curl -s -w "\n" http://prometheus:9090/api/v1/alertsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
docker start appShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
For teachers
Tip
This lesson is a small example of a lesson worked in a lab: the steps and the Run buttons are the lesson, and the lab is each student's own copy. The lab is four services and two named volumes, and nothing here marks itself: the student hands in the rule, a screenshot of it firing and a Grafana panel, with a sentence on what someone on call would do. The rule is in a file on the Prometheus volume, so it is still there after the lab is stopped and started. Copy it into your own module and change it, or build one from scratch with the Lab menu in the lesson builder.