What a latency budget is
Latency is the time a user waits for an answer. A latency budget is the most time you allow, shared out between the steps of one request.
The module on cost, tracing and reliability shows percentiles and a timeout. This lesson goes further: a budget for the whole request, and what streaming does and does not change.
By the end of this lesson
Note
I can set a latency budget and say what streaming changes.
Key words
| Word | What it means | Example |
|---|---|---|
| Latency | the time a user waits | |
| Time to first token | the wait before the first words appear | 600 ms |
| Total time | the wait until the last word | |
| Streaming | sending words as they are made, not all at the end | |
| Budget | the most time you allow, shared between steps | |
| Millisecond | a thousandth of a second |
The cells students fill in are left blank in this preview.
Two times: first words and last word
A reply takes some time before the first word, then a little more for each word after it. The first time is the time to first token. The last is the total time.
Total time is the time to first token plus the time for each token times the number of tokens.
Worked example: two replies
| Reply | To first token | Per token | Tokens | Total |
|---|---|---|---|---|
| Short | 600 | 20 | 30 | 1200 |
| Long | 600 | 20 | 300 | 6600 |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think streaming makes a reply faster. It does not. The last word arrives at exactly the same time. What changes is when the user first sees words, so the wait feels shorter.
Where the wait is felt
Without streaming, the user sees nothing until the total time. With streaming, they see words after the time to first token and can start reading.
For a long reply the difference is large. In the example, the wait for first words falls from 6600 to 600 milliseconds, a saving of 6000.
What streaming costs you
Important
Words are shown before the whole reply is checked. A check that needs the full text, such as a safety check, cannot run first. Decide how you will handle that before you turn streaming on.
Share the budget between steps
A request has more steps than the model. It waits in a queue, fetches documents and checks the reply. Add the times in order.
The first step where the running total goes over the budget is the one to fix first, or to cut.
Worked example: a budget of 1000
| Step | Time | Running total | Over 1000? |
|---|---|---|---|
| Queue | 150 | 150 | No |
| Retrieval | 400 | 550 | No |
| Model first token | 600 | 1150 | Yes, first over |
| Safety check | 200 | 1350 | Yes, 350 over |
The cells students fill in are left blank in this preview.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the latency script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'function total(r) { return r.ttft + r.perToken * r.tokens; }' 'function first(r) { return r.streamed ? r.ttft : total(r); }' 'const replies = [' ' { name: \"short\", ttft: 600, perToken: 20, tokens: 30, streamed: false },' ' { name: \"long\", ttft: 600, perToken: 20, tokens: 300, streamed: false },' ' { name: \"long-streamed\", ttft: 600, perToken: 20, tokens: 300, streamed: true },' '];' > latency.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the latency script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const parts = [[\"queue\", 150], [\"retrieval\", 400], [\"model first token\", 600], [\"safety check\", 200]];' 'function budget(limit) {' ' let sum = 0;' ' let over = \"none\";' ' for (const p of parts) {' ' sum = sum + p[1];' ' if (sum > limit && over === \"none\") over = p[0];' ' }' ' return \"total \" + sum + \", left \" + (limit - sum) + \", first over: \" + over;' '}' >> latency.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the latency script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' for (const r of replies) console.log(r.name + \": first \" + first(r) + \", total \" + total(r));' ' console.log(\"budget 1000: \" + budget(1000));' ' console.log(\"budget 2000: \" + budget(2000));' ' const started = Date.now();' ' const reply = await ask([{ role: \"user\", content: \"In one short sentence, why do users like to see words appear early?\" }], \"They can start reading at once, so the wait feels shorter.\", 60);' ' console.log(\"model said: \" + reply.text);' ' console.log(\"measured wait for the whole reply: \" + (reply.recorded ? \"not measured\" : Date.now() - started + \" ms\"));' '}' 'main();' >> latency.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the latency sums
docker exec -w /home/node/project agent node latency.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| short: first 1200, total 1200 | No streaming: the first text is the total |
| long-streamed: first 600, total 6600 | Streaming: first words early, same total |
| budget 1000: total 1350, left -350, first over: model first token | The step to fix |
| model said: ... | A sentence from the lab. A recorded sample if no model answers |
| measured wait for the whole reply: ... | A real time in ms, or not measured for a recorded sample |
The cells students fill in are left blank in this preview.