Two numbers that are easy to fake
Measuring LLM latency means timing how long a user waits for the first token and how fast the rest arrive. The two usual numbers are time to first token and tokens per second.
Both numbers are easy to make look better than they are. Where the clock starts and which tokens are counted change them more than any speed-up does.
By the end of this lesson
Note
I can work out time to first token and tokens per second from token times.
Key words
| Word | What it means |
|---|---|
| Latency | how long somebody waits |
| Time to first token | the wait from sending a request to the first token |
| Tokens per second | output tokens divided by the seconds they took |
| Inter-token gap | the time between two tokens in a reply |
| Warm-up | a first run that loads things and is slower |
| Median | the middle value once the values are sorted |
The cells students fill in are left blank in this preview.
Start the clock when the request arrives
Time to first token is what a user feels as "is it working?". It is the time from the request arriving to the first token. It includes any wait in a queue, then the prefill pass, then the first decode step.
A server that starts its clock when it begins work hides the queue. On a busy server the queue can be most of the wait.
Work the numbers out from one request
docker exec jupyter python3 steps/latency.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: one request on a simulated clock
| Measure | Value | Honest? |
|---|---|---|
| Time to first token, from arrival | 345 ms | Yes |
| Time to first token, from the start of work | 225 ms | No: hides 120 ms of queue |
| Decode speed, after the first token | 40.0 tokens a second | Yes, for decode |
| Whole request, output tokens only | 31.8 tokens a second | Yes, for the user |
| Prompt counted as output | 286.6 tokens a second | No |
The cells students fill in are left blank in this preview.
Count only the tokens the model wrote
Decode speed is the output tokens after the first, divided by the time from the first token to the last. Here that is 49 tokens in 1.225 seconds, 40.0 a second.
The whole request's speed divides all 50 output tokens by the time from arrival, 31.8 a second. Both are honest and answer different questions. Say which one you report.
A common mistake
Warning
Many people count the prompt's tokens in tokens per second, because the model did process them. That turned 31.8 into 286.6 here. Prompt tokens are read in one fast pass. Report output tokens per second, and give the prompt length beside it.
Real clocks need warm-up and repeats
A real timer gives a different number every run. The first run is often slow, because code and data are loaded. One run can also be hit by another program.
So run once to warm up and throw that away. Repeat several times and report the median, which one odd run cannot drag. Name the machine and the settings, since another machine gives other numbers.
Try it: mean against median
# Eight timings of the same request, in milliseconds. The first one was cold.
runs = [41.0, 12.5, 12.1, 13.0, 12.4, 30.2, 12.6, 12.3]
total = 0
for one in runs:
total = total + one
print("mean of all runs:", round(total / len(runs), 2))
warm = sorted(runs[1:])
print("middle of the warm runs:", warm[len(warm) // 2])
# Change 30.2 to 300. Which number moves, and which stays?
Time the tiny model for real
In the lab, open JupyterLab and the notebook start.ipynb, then run the cell under Real timings. It times prefill, one decode step and three batch sizes with a real clock. It warms up first, repeats seven times and prints the middle value.
Your numbers will differ from your neighbour's and from one run to the next. That is why the steps scripts count work instead.