One model, many users at once
Continuous batching is how a language model server answers many users at once. It decodes their requests together and slots a new one in as soon as another finishes.
A server that answers one user at a time wastes most of its memory reads. Batching is the biggest single reason a shared model is affordable.
By the end of this lesson
Note
I can schedule requests with static and continuous batching and compare the two.
Key words
| Word | What it means |
|---|---|
| Request | one user's prompt and the reply it needs |
| Batch | requests decoded together in one pass |
| Slot | a place for one request in the batch |
| Static batching | a new batch starts only when the last one ends |
| Continuous batching | a free slot is filled at the next step |
| Throughput | tokens made per second by the whole server |
The cells students fill in are left blank in this preview.
A batch shares one read of the weights
A decode pass reads every weight to make one token. If eight requests are decoded in the same pass, the weights are read once and used eight times.
Each request still needs its own sums and its own KV cache. Only the read of the weights is shared, and on a big model that read is the slow part.
Count the reads at four batch sizes
docker exec jupyter python3 steps/share.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: eight decode passes at each size
| Batch | Tokens made | Weight bytes read per token | Multiply-adds per token |
|---|---|---|---|
| 1 | 8 | 12,108 | 3,427 |
| 2 | 16 | 6,054 | 3,427 |
| 4 | 32 | 3,027 | 3,427 |
| 8 | 64 | 1,513 | 3,427 |
The cells students fill in are left blank in this preview.
What a CPU shows
Note
This model is 12 KB, so its weights sit in the CPU's fast cache and reading them is cheap. Its real timings, in the notebook's Real timings cell, gain far less than the batch size. A big model on a GPU gains much more, because there the reads are the slow part.
Static batching waits for the slowest
Requests want replies of different lengths. With static batching, a batch starts together and the server takes no new request until every one in it has finished.
A request wanting 2 tokens finishes early, and its slot sits empty while a long one carries on. New arrivals wait outside the whole time.
Schedule six requests both ways
docker exec jupyter python3 steps/batch.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: three slots, six requests
| Request | Arrives at step | Tokens | Finishes, static | Finishes, continuous |
|---|---|---|---|---|
| A | 0 | 8 | 8 | 8 |
| B | 0 | 2 | 2 | 2 |
| C | 0 | 3 | 3 | 3 |
| D | 1 | 2 | 10 | 4 |
| E | 2 | 6 | 14 | 9 |
| F | 3 | 2 | 10 | 6 |
The cells students fill in are left blank in this preview.
Continuous batching refills a slot at once
With continuous batching, the server looks for a free slot before every step. When B finished at step 2, D took its slot at the next step instead of waiting for A.
The batch is never emptied and refilled as a whole. Each request joins and leaves on its own, so short requests stop waiting behind long ones.
A common mistake
Warning
Many people think batching makes every reply come back sooner. It raises the tokens a second of the whole server. A single request may wait for a slot, and a bigger batch can make each step a little slower. Measure both throughput and each user's wait.
Try it: memory decides the slots
# How many requests fit beside the weights? Each needs its own KV cache.
memory_gb = 24
weights_gb = 14
cache_gb_per_request = 1
slots = (memory_gb - weights_gb) // cache_gb_per_request
print("requests that fit at once:", slots)
# Quantise the weights to 7 GB. How many fit now?