Why does a chatbot type its answer?
When you ask a chatbot something, the answer appears a word at a time. That is not a show: the model really makes one token, then the next. This lab gives you a tiny language model to watch doing it, and to make it faster in the ways real servers do. It reads characters and is far too small to be useful.
By the end of this lesson
Note
I can explain how a language model writes a reply one token at a time.
Key words
- Inference: running a trained model to get an answer.
- Token: one piece of text the model reads or writes. Here, one character.
- Prefill: the first pass, which reads the whole prompt.
- Decode: each later pass, which makes one new token.
- KV cache: the keys and values of earlier tokens, kept so they are not worked out again.
- Quantise: store numbers in fewer bytes, with a scale to turn them back.
Starter: what you already know
Note
Answer in your head. 1. What does a loop do? 2. If you read a long page twice, does it take more or less time than reading it once? 3. Which takes less room, the number 0.3014 or the number 30? Answers: it repeats a step; more; 30.
Before you start
Note
Press Start the lab on the bar at the top of this lesson and wait until it says it is running. There is one container, jupyter, and the terminal reaches its Python with docker exec jupyter, which runs one command inside it. The JupyterLab link opens its web page, port 8888. Each Run button below types its command into your lab's terminal, one after another in the same session. To watch them, or to type your own, press Open terminal on the bar.
The model was trained when the lab started
ls lists the work folder: the model tiny.py, the file of weights model.npz, the notebook, the folder steps and train.log. That log is what training printed. The loss, how surprised the model is by the next character, fell from 2.944 to 0.317 over 2000 steps, on 242 characters of text with 19 different characters.
docker exec jupyter lsShown here, not run. Press Start lab to type it into your own lab’s terminal.
docker exec jupyter cat train.logShown here, not run. Press Start lab to type it into your own lab’s terminal.
One pass reads the prompt, then one pass a token
The script gives the model the prompt "the cat sat ". The first line says it read all 12 characters in one pass: that is prefill. Then every line is one decode pass that chooses one character, until the text reads "the cat sat on the mat".
docker exec jupyter python3 steps/generate.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
A KV cache saves reading the text again
The script writes 32 characters twice, once reading the whole text again on every pass and once with a KV cache. The text is the same both ways. Without the cache the model worked out 880 rows of keys and values; with it, 43. The cache did 8.8 times less work.
docker exec jupyter python3 steps/cache.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
Check
Tip
Why did the cache not change the text? Answer: it stores the same keys and values the model would work out again, so every choice is the same. It changes the work, not the answer.
Batching shares the work between users
A server answers several people at once in slots. The script schedules six requests in three slots two ways. Static batching waits until a whole batch has finished: 14 steps, slots 55% in use. Continuous batching fills a free slot at the next step: 9 steps, 85% in use.
docker exec jupyter python3 steps/batch.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
A small model guesses, the big model checks
In speculative decoding, a tiny draft model guesses four characters and the model checks them in one pass, keeping the guesses up to the first wrong one. Each line is one round. At the end, the text is the same as plain decoding (True), made in 13 passes instead of 32.
docker exec jupyter python3 steps/speculate.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
Int8 stores each weight in one byte
The script rounds every weight to a whole number from -127 to 127 with a scale for each row. The weights shrink from 12108 bytes to 3543. The model still chooses the same next character at 64 of 64 places, and writes the same reply.
docker exec jupyter python3 steps/quantise.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
A common mistake
Warning
Many people think these tricks make the model cleverer. None of them changes what it writes: the cache, batching and speculative decoding give exactly the same text, and int8 gave the same text here. They change the cost: how much work, how much memory and how long. Always check the answer stayed the same.
The same work in a notebook
Open JupyterLab from the bar and double-click start.ipynb. Click a cell and press Shift and Enter to run it. The cell under Real timings times the model with a real clock, so its numbers change from run to run. The last cell is where your own question goes.
Exit check
Tip
Answer without looking back. 1. What is the first pass, which reads the prompt, called? 2. How many tokens does a decode pass make? 3. What did the KV cache save? 4. Did int8 change the reply here? Answers: prefill; one; working out the keys and values again, 880 rows against 43; no.
For teachers
Tip
This lesson is a small example of a lesson worked in a lab: the steps and the Run buttons are the lesson, and the lab is each student's own copy. The lab is one service, JupyterLab with a tiny character model written in NumPy and trained at start, and nothing here marks itself: the student hands in a notebook of their own, downloaded from JupyterLab, and a sentence on what it showed. Key learning points: 1. A model writes one token a pass after one prefill pass. 2. A KV cache, batching and speculative decoding change the cost and not the text. 3. Int8 shrinks the weights to about a quarter. Common mistake: thinking these tricks change the answer. Equipment: the lab and a browser. Tip: ask students to predict the cache's saving before they run it. The module How language model inference works, on a CPU, is worked in the same lab. Copy it into your own module and change it, or build one from scratch with the Lab menu in the lesson builder.