The players noticed first
A college games club runs a small server for an online game. One evening the game feels slow. The first anyone hears of it is a stream of angry messages in the group chat.
If the server had been writing down a few numbers every few seconds, the club could have spotted the problem before the players did.
By the end of this lesson
Note
I can tell when a service is unwell.
Key words
| Word | What it means | Example |
|---|---|---|
| Metric | a number about a service, measured again and again | |
| Counter | a metric that only goes up | nginx_http_requests_total |
| Rate | how much a counter grows each second | |
| Average | the total divided by how many values there are | |
| Percentile | the value that a share of the measurements stay at or under | |
| Alert | a message sent when a metric stays wrong |
The cells students fill in are left blank in this preview.
Before you start
Note
Press Start the lab on the bar at the top and wait until it says it is ready. Your terminal reaches each part of the lab by its name: app, exporter and prometheus.
First, ask the service itself
The service you will watch is called app. It is a web server that answers every visit with one line of text.
It also has a status page on another port. That page counts the visits it has answered. The second Run reads it.
Visit the app
curl -s http://app:8080/Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Read the app's status page
curl -s http://app:8081/stub_statusShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Prometheus writes the numbers down
Reading a page by hand is fine once. To watch all day you need a program. A helper called the exporter reads the status page and turns it into metrics.
Prometheus asks the exporter every five seconds and keeps each number with the time. Its metric up is 1 for each part that answers and 0 for one that does not.
Which parts answer?
curl -s -G -w "\n" http://prometheus:9090/api/v1/query --data-urlencode 'query=up'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A rate turns a counter into a speed
A counter only goes up. On its own, "68 visits" says little. Was that in a minute or in a week?
The rate is how much the counter grew, divided by the seconds it took. Read the counter, send fifty visits, then ask Prometheus for the rate over the last minute.
Read the visits counter
curl -s -G -w "\n" http://prometheus:9090/api/v1/query --data-urlencode 'query=nginx_http_requests_total'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Send fifty visits
curl -s -o /dev/null "http://app:8080/?n=[1-50]"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Ask for the rate per second
curl -s -G -w "\n" http://prometheus:9090/api/v1/query --data-urlencode 'query=rate(nginx_http_requests_total[1m])'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Worked example: a rate by hand
| Reading | Counter | Seconds later |
|---|---|---|
| Before the visits | 16 | 0 |
| After the visits | 68 | 8 |
| Growth and time | 68 minus 16 is 52 | 8 seconds |
| Rate | 52 divided by 8 is 6.5 | per second |
The cells students fill in are left blank in this preview.
The slow few hide in the average
Ten players wait for the game server to reply. Nine waits are under 60 milliseconds and one is 900. The average is 125.1, a wait that no player had.
A percentile is better. Sort the waits. The 90th percentile is the ninth of ten, 52. Nine players in ten waited 52 milliseconds or less.
Worked example: ten waits in milliseconds
| Measure | Value | What it says |
|---|---|---|
| The waits, sorted | 30, 32, 35, 36, 40, 40, 41, 45, 52, 900 | one player waited almost a second |
| Average | 125.1 | a wait nobody had |
| Median, the 50th percentile | 40 | half waited this long or less |
| 90th percentile | 52 | nine in ten waited this long or less |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think the average wait shows how users feel. One very slow wait pulls the average up, and many fast ones hide it. Read the median for the usual user and a high percentile for the unlucky ones.
The same numbers in Python
waits = [30, 32, 35, 36, 40, 40, 41, 45, 52, 900]
total = 0
for wait in waits:
total = total + wait
print("average", total / len(waits))
print("median", waits[4])
print("90th percentile", waits[8])
# Change 900 to 60 and press Run. Which number moves most?Going further: one minute or five
Send three hundred visits with n=[1-300] in the same curl. Then ask for rate(nginx_http_requests_total[1m]) and rate(nginx_http_requests_total[5m]). Ask once straight away and once two minutes later.
Write down the four numbers. Which rate jumps more, and which stays raised for longer?