Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Measuring LLM latency and throughput

After this lesson: I can work out time to first token and tokens per second from token times.

60 min lesson1 commandLLM inference lab: a tiny language model on a CPU, with a KV cache, batching and int8

Two numbers that are easy to fake

Measuring LLM latency means timing how long a user waits for the first token and how fast the rest arrive. The two usual numbers are time to first token and tokens per second.

Both numbers are easy to make look better than they are. Where the clock starts and which tokens are counted change them more than any speed-up does.

By the end of this lesson

Note

I can work out time to first token and tokens per second from token times.

Key words

WordWhat it means
Latencyhow long somebody waits
Time to first tokenthe wait from sending a request to the first token
Tokens per secondoutput tokens divided by the seconds they took
Inter-token gapthe time between two tokens in a reply
Warm-upa first run that loads things and is slower
Medianthe middle value once the values are sorted

The cells students fill in are left blank in this preview.

Start the clock when the request arrives

Time to first token is what a user feels as "is it working?". It is the time from the request arriving to the first token. It includes any wait in a queue, then the prefill pass, then the first decode step.

A server that starts its clock when it begins work hides the queue. On a busy server the queue can be most of the wait.

Work the numbers out from one request

Command
docker exec jupyter python3 steps/latency.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: one request on a simulated clock

MeasureValueHonest?
Time to first token, from arrival345 msYes
Time to first token, from the start of work225 msNo: hides 120 ms of queue
Decode speed, after the first token40.0 tokens a secondYes, for decode
Whole request, output tokens only31.8 tokens a secondYes, for the user
Prompt counted as output286.6 tokens a secondNo

The cells students fill in are left blank in this preview.

Count only the tokens the model wrote

Decode speed is the output tokens after the first, divided by the time from the first token to the last. Here that is 49 tokens in 1.225 seconds, 40.0 a second.

The whole request's speed divides all 50 output tokens by the time from arrival, 31.8 a second. Both are honest and answer different questions. Say which one you report.

A common mistake

Warning

Many people count the prompt's tokens in tokens per second, because the model did process them. That turned 31.8 into 286.6 here. Prompt tokens are read in one fast pass. Report output tokens per second, and give the prompt length beside it.

Real clocks need warm-up and repeats

A real timer gives a different number every run. The first run is often slow, because code and data are loaded. One run can also be hit by another program.

So run once to warm up and throw that away. Repeat several times and report the median, which one odd run cannot drag. Name the machine and the settings, since another machine gives other numbers.

Try it: mean against median

Code: python
# Eight timings of the same request, in milliseconds. The first one was cold.
runs = [41.0, 12.5, 12.1, 13.0, 12.4, 30.2, 12.6, 12.3]
total = 0
for one in runs:
    total = total + one
print("mean of all runs:", round(total / len(runs), 2))
warm = sorted(runs[1:])
print("middle of the warm runs:", warm[len(warm) // 2])
# Change 30.2 to 300. Which number moves, and which stays?

Time the tiny model for real

In the lab, open JupyterLab and the notebook start.ipynb, then run the cell under Real timings. It times prefill, one decode step and three batch sizes with a real clock. It warms up first, repeats seven times and prints the middle value.

Your numbers will differ from your neighbour's and from one run to the next. That is why the steps scripts count work instead.

About this lab and its source: LLM inference lab: a tiny language model on a CPU, with a KV cache, batching and int8

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.