Fifty people ask at once
A model server is a program that holds a model in memory and answers many people. It puts requests in a queue and groups them into batches.
That matters because one chip answering one person at a time leaves everyone else waiting. Batching is how a server stays fast when the room is full.
By the end of this lesson
Note
I can describe how a server turns many requests into batches.
Key words
| Word | What it means |
|---|---|
| Queue | requests waiting in the order they came |
| Batch | requests answered together |
| Wait | time from arriving to starting |
| Time to first word | wait plus the time to make the first token |
| Throughput | how many requests finish in a set time |
The cells students fill in are left blank in this preview.
A queue feeds the chip
Requests join a queue. When the chip is free, the server takes the requests that have arrived and runs them together as one batch.
In our model, a batch costs 2 ticks to start plus the work of its longest request. So adding a short request to a batch costs almost nothing.
The same five requests, two batch sizes
| Batch size | Waits (a, b, c, d, e) | Average wait |
|---|---|---|
| 1 | 0 5 9 16 14 | 8.80 |
| 3 | 0 5 4 3 5 | 3.40 |
The cells students fill in are left blank in this preview.
Two speeds: first word and words a second
Time to first word is the wait plus the time to make the first token. It is what a person feels as lag. Throughput is how many requests finish in a set time. It is what the owner pays for.
A batch raises throughput. But a short request in a batch ends only when the longest one ends, so one very long request can slow the rest.
A common mistake
Warning
Many people think a bigger batch is always better. A bigger batch raises throughput, but each batch lasts as long as its longest request. Choose the size by watching both the waits and the throughput.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the queue simulator, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const reqs = [[\"a\", 0, 4], [\"b\", 1, 3], [\"c\", 2, 6], [\"d\", 3, 2], [\"e\", 9, 3]];' 'function simulate(list, maxBatch) {' ' const out = [];' ' let time = 0;' ' let next = 0;' ' while (next < list.length) {' ' if (list[next][1] > time) time = list[next][1];' ' const batch = [];' ' while (next < list.length && batch.length < maxBatch && list[next][1] <= time) {' ' batch.push(list[next]);' ' next++;' ' }' > serve.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the queue simulator, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' ' const end = time + 2 + Math.max(...batch.map(function (r) { return r[2]; }));' ' batch.forEach(function (r) { out.push({ id: r[0], wait: time - r[1], end: end }); });' ' time = end;' ' }' ' return out;' '}' >> serve.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the queue simulator, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function report(maxBatch) {' ' const rows = simulate(reqs, maxBatch);' ' const waits = rows.map(function (r) { return r.wait; });' ' const mean = waits.reduce(function (a, b) { return a + b; }, 0) / waits.length;' ' console.log(\"batch \" + maxBatch + \": waits \" + waits.join(\" \") + \", average \" + mean.toFixed(2) + \", finished at \" + rows[rows.length - 1].end);' '}' >> serve.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the queue simulator, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' report(1);' ' report(3);' ' const q = \"In one short sentence, why does a server group requests into a batch?\";' ' const sample = \"A batch lets the chip work on several requests at once, so more people are served in the same time.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> serve.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the queue simulator
docker exec -w /home/node/project agent node serve.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| batch 1: waits 0 5 9 16 14, average 8.80 ... | One at a time: long waits |
| batch 3: waits 0 5 4 3 5, average 3.40 ... | Grouped: shorter waits |
| finished at 28 and 19 | The whole queue clears sooner |
| model said: ... | A sentence. The words differ each run |
The cells students fill in are left blank in this preview.