The shop is slow on Friday nights
An online shop feels slow every Friday evening. The owner asks how slow, and since when. Nobody can say.
If the shop had written down a few numbers every few seconds, the answer would be a graph. Measuring comes first. Fixing comes after.
By the end of this lesson
Note
I can read a metric and say what it tells me about a service.
Key words
| Word | What it means | Example |
|---|---|---|
| Metric | a number about a service, measured again and again | |
| Counter | a metric that only goes up | nginx_http_requests_total |
| Gauge | a metric that goes up and down | nginx_connections_active |
| Rate | how fast a counter grows, per second | |
| Percentile | the value that a share of the measurements stay at or under | |
| Alert | a message sent when a metric looks wrong |
The cells students fill in are left blank in this preview.
A counter climbs and a gauge wobbles
A metric is a number a service shares about itself. In the lab, a small helper called an exporter reads nginx's status page and turns it into metrics. A collector called Prometheus asks the exporter every five seconds and keeps each number with the time.
A counter only grows: requests served since the start. A gauge moves both ways: connections open right now.
The first Run asks Prometheus which targets answer.
Which targets answer?
curl -s -G -w "\n" http://prometheus:9090/api/v1/query --data-urlencode 'query=up'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Ask for a gauge
curl -s -G -w "\n" http://prometheus:9090/api/v1/query --data-urlencode 'query=nginx_connections_active'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Ask for a counter
curl -s -G -w "\n" http://prometheus:9090/api/v1/query --data-urlencode 'query=nginx_http_requests_total'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A rate turns a counter into a speed
A counter that reads 72 says little. Was it 72 in a day or in a minute?
The rate is the growth divided by the seconds. If a counter went from 19 to 72 in 8 seconds, it grew by 53, so the rate is 53 divided by 8.
Run the first command to send fifty visits to the web server. Then ask Prometheus for the rate over the last minute.
The rate by hand
before = 19
after = 72
seconds = 8
print((after - before) / seconds)
# Change seconds to 4. Is the service busier or quieter?Send fifty visits
curl -s -o /dev/null "http://app:8080/?n=[1-50]"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Ask for the rate per second
curl -s -G -w "\n" http://prometheus:9090/api/v1/query --data-urlencode 'query=rate(nginx_http_requests_total[1m])'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
A percentile shows the slow few
Ten customers wait 20, 22, 25, 25, 28, 28, 31, 35, 40 and 950 milliseconds. The average is 120.4. No customer waited that long.
The 90th percentile is the time that nine in ten customers wait no longer than. Sort the times and take the ninth: 40.
The lab's server does not publish times, so you work this one by hand.
Worked example: ten waiting times
| Measure | Value | What it says |
|---|---|---|
| Average | 120.4 ms | A number no customer lived |
| Median, the 50th percentile | 28 ms | Half waited this long or less |
| 90th percentile | 40 ms | Nine in ten waited this long or less |
| Slowest | 950 ms | One customer waited nearly a second |
The cells students fill in are left blank in this preview.
The same numbers in code
times = [20, 22, 25, 25, 28, 28, 31, 35, 40, 950]
total = 0
for time in times:
total = total + time
print("average", total / len(times))
print("median", times[4])
print("90th", times[8])
# Change 950 to 45. Which number moves most?A common mistake
Warning
Many people think the average time shows how users feel. A few very slow requests drag it up, and many fast ones hide them. Read the median for the usual customer and a high percentile for the unlucky one.
An alert is a rule on a metric
People cannot watch a graph all day. An alert is a rule that watches a metric for them. It says "if this stays wrong for this long, tell someone".
The time matters. A spike for one second should not wake anybody. Prometheus keeps its rules in a list. A new lab copy has none yet, so the next Run shows an empty list.
List the rules Prometheus knows
curl -s -w "\n" http://prometheus:9090/api/v1/rulesShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Going further
Send three hundred visits with n=[1-300] in the same curl. Then ask Prometheus for rate(nginx_http_requests_total[1m]) and for rate(nginx_http_requests_total[5m]), once straight away and once two minutes later.
Write down the four numbers. Explain why the five-minute rate is lower at first and stays raised for longer.