Free module
How language model inference works, on a CPU
Learn how LLM inference works hands on: prefill and decode, the KV cache, continuous batching, speculative decoding, int8 and latency.
The lessons
Each lesson is free and interactive. Press Start lab to run its commands in a live lab and to sit its quizzes and tasks, with no account needed.
1 · Container lab
Prefill and decode in LLM inference
I can explain why decode is limited by reading memory while prefill is not.
Read the lesson2 · Container lab
How a KV cache speeds up generation
I can work out how much work a KV cache saves and how much memory it costs.
Read the lesson3 · Container lab
Batching and continuous batching
I can schedule requests with static and continuous batching and compare the two.
Read the lesson4 · Container lab
How speculative decoding works
I can decide which draft tokens a target model keeps in speculative decoding.
Read the lesson5 · Container lab
Int8 quantisation by hand
I can quantise a row of weights to int8 and measure the error.
Read the lesson6 · Container lab
Measuring LLM latency and throughput
I can work out time to first token and tokens per second from token times.
Read the lesson
Worked in a real lab
Every lesson of this module is worked in a lab started for you alone. The labs it uses:
Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.