Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

Batching and continuous batching

After this lesson: I can schedule requests with static and continuous batching and compare the two.

60 min lesson2 commandsLLM inference lab: a tiny language model on a CPU, with a KV cache, batching and int8

One model, many users at once

Continuous batching is how a language model server answers many users at once. It decodes their requests together and slots a new one in as soon as another finishes.

A server that answers one user at a time wastes most of its memory reads. Batching is the biggest single reason a shared model is affordable.

By the end of this lesson

Note

I can schedule requests with static and continuous batching and compare the two.

Key words

WordWhat it means
Requestone user's prompt and the reply it needs
Batchrequests decoded together in one pass
Slota place for one request in the batch
Static batchinga new batch starts only when the last one ends
Continuous batchinga free slot is filled at the next step
Throughputtokens made per second by the whole server

The cells students fill in are left blank in this preview.

A batch shares one read of the weights

A decode pass reads every weight to make one token. If eight requests are decoded in the same pass, the weights are read once and used eight times.

Each request still needs its own sums and its own KV cache. Only the read of the weights is shared, and on a big model that read is the slow part.

Count the reads at four batch sizes

Command
docker exec jupyter python3 steps/share.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: eight decode passes at each size

BatchTokens madeWeight bytes read per tokenMultiply-adds per token
1812,1083,427
2166,0543,427
4323,0273,427
8641,5133,427

The cells students fill in are left blank in this preview.

What a CPU shows

Note

This model is 12 KB, so its weights sit in the CPU's fast cache and reading them is cheap. Its real timings, in the notebook's Real timings cell, gain far less than the batch size. A big model on a GPU gains much more, because there the reads are the slow part.

Static batching waits for the slowest

Requests want replies of different lengths. With static batching, a batch starts together and the server takes no new request until every one in it has finished.

A request wanting 2 tokens finishes early, and its slot sits empty while a long one carries on. New arrivals wait outside the whole time.

Schedule six requests both ways

Command
docker exec jupyter python3 steps/batch.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: three slots, six requests

RequestArrives at stepTokensFinishes, staticFinishes, continuous
A0888
B0222
C0333
D12104
E26149
F32106

The cells students fill in are left blank in this preview.

Continuous batching refills a slot at once

With continuous batching, the server looks for a free slot before every step. When B finished at step 2, D took its slot at the next step instead of waiting for A.

The batch is never emptied and refilled as a whole. Each request joins and leaves on its own, so short requests stop waiting behind long ones.

A common mistake

Warning

Many people think batching makes every reply come back sooner. It raises the tokens a second of the whole server. A single request may wait for a slot, and a bigger batch can make each step a little slower. Measure both throughput and each user's wait.

Try it: memory decides the slots

Code: python
# How many requests fit beside the weights? Each needs its own KV cache.
memory_gb = 24
weights_gb = 14
cache_gb_per_request = 1
slots = (memory_gb - weights_gb) // cache_gb_per_request
print("requests that fit at once:", slots)
# Quantise the weights to 7 GB. How many fit now?

About this lab and its source: LLM inference lab: a tiny language model on a CPU, with a KV cache, batching and int8

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.