Say where the time goes
Latency is the time from a request to its answer. For a model call it is two numbers: the wait for the first token, and the wait for the last. It matters because users judge a feature by its slowest moments, not by its average.
A feature that answers in a second for nineteen users and in eight seconds for the twentieth feels broken to the twentieth.
By the end of this lesson
Note
I can say where the time goes in a model call and set a timeout from recorded durations.
Key words
| Word | What it means |
|---|---|
| Latency | the time a request takes to be answered |
| First token | the first piece of a streamed reply |
| Percentile | the value below which a share of requests fall |
| Tail | the slowest few requests |
| Timeout | the longest wait before giving up |
| Streaming | sending a reply in pieces as it is made |
The cells students fill in are left blank in this preview.
Percentiles show the tail
An average hides the slow requests. Sort the durations instead. The median (p50) is the middle one. p95 is the value that 95 percent of requests are at or under. The few above it are the tail.
Use nearest rank: with n values, take the one at place p percent of n, rounded up, counting from 1. It needs no interpolation, so every tool gives the same answer.
Worked example: twenty durations
| Measure | Place in 20 | Value |
|---|---|---|
| p50 | 10th | 900 ms |
| p90 | 18th | 2300 ms |
| p95 | 19th | 4200 ms |
| p99 | 20th | 7800 ms |
| Average | none | 1493 ms |
The cells students fill in are left blank in this preview.
Set a timeout from the data
A timeout is the longest you wait. Too short and you cut good replies. Too long and a stuck call holds a user and a connection. Take a high percentile such as p95, add a margin, and round.
Streaming hides delay: here half the users see the first token within 230 ms, while the rest arrives. Measure the first token and the whole reply separately, and give each its own limit.
A common mistake
Warning
Many people set a timeout from the average, because it is one easy number. Then good replies are cut: here a 1493 ms timeout would cut 4 of 20. Set it from a high percentile of recorded durations, and check how many it would cut.
Write the latency report, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// Twenty recorded, invented durations in milliseconds: the whole reply, and the first token.' 'const TOTAL = [820, 910, 760, 1500, 880, 940, 830, 4200, 900, 870, 1020, 790, 850, 2300, 910, 880, 960, 840, 7800, 900];' 'const FIRST = [210, 240, 200, 300, 220, 250, 215, 500, 230, 225, 260, 205, 222, 380, 240, 228, 255, 218, 900, 232];' > latency4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the latency report, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// Nearest rank: sort, then take the value at position ceil(p percent of n).' 'function percentile(values, p) {' ' const sorted = values.slice().sort(function (a, b) { return a - b; });' ' const rank = Math.ceil(p * sorted.length / 100);' ' return sorted[rank - 1];' '}' 'function average(values) {' ' return Math.round(values.reduce(function (sum, v) { return sum + v; }, 0) / values.length);' '}' >> latency4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the latency report, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' '// A timeout: the 95th percentile plus a 20 percent margin, rounded up to 100 ms.' 'function timeoutFor(values) {' ' return Math.ceil(percentile(values, 95) * 120 / 100 / 100) * 100;' '}' 'function over(values, limit) {' ' return values.filter(function (v) { return v > limit; }).length;' '}' >> latency4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the latency report, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const limit = timeoutFor(TOTAL);' 'console.log(\"whole reply: p50 \" + percentile(TOTAL, 50) + \", p90 \" + percentile(TOTAL, 90) + \", p95 \" + percentile(TOTAL, 95) + \", p99 \" + percentile(TOTAL, 99) + \" ms\");' 'console.log(\"average \" + average(TOTAL) + \" ms, slowest \" + Math.max(...TOTAL) + \" ms\");' 'console.log(\"first token: p50 \" + percentile(FIRST, 50) + \", p95 \" + percentile(FIRST, 95) + \" ms\");' 'console.log(\"timeout \" + limit + \" ms cuts \" + over(TOTAL, limit) + \" of \" + TOTAL.length);' 'console.log(\"a 2000 ms timeout would cut \" + over(TOTAL, 2000) + \" of \" + TOTAL.length);' >> latency4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the report
docker exec -w /home/node/project agent node latency4.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
These durations are recorded
Nothing here makes a call slow on purpose. The twenty durations are recorded and invented, so every student gets the same numbers. A real system reads them from its traces, as in the first lesson.
The Run calls no model, so it needs no helper and has nothing to fall back on.
Output of the Run
| Line | What it shows |
|---|---|
| whole reply: p50 900, p90 2300, p95 4200, p99 7800 ms | The shape of the whole reply |
| average 1493 ms, slowest 7800 ms | The average sits between the two groups |
| first token: p50 230, p95 500 ms | Streaming shows a first token far earlier |
| timeout 5100 ms cuts 1 of 20 | p95 plus 20 percent, rounded up to 100 |
| a 2000 ms timeout would cut 3 of 20 | A guess cuts good replies |
The cells students fill in are left blank in this preview.