LLM inference lab: a tiny language model on a CPU, with a KV cache, batching and int8

This lab gives each student a JupyterLab of their own with a tiny language model written in NumPy, short enough to read in full: it reads characters, has one attention layer and keeps a KV cache. It is trained when the lab starts, in under a second, from a fixed text and a fixed seed, so every copy has the same model. Short scripts show how inference works and how servers make it faster, and print the same every time because they count work instead of timing it.

The model is far too small to be useful, and the lab says so: it shows the mechanism only. There is no GPU, nothing is downloaded and no AI service is called. It is not an attack lab either: every copy has no route to the internet and none to any other student.

What students learn

How a language model answers: one prefill pass reads the prompt, then one decode pass makes each new token, and why decode is limited by reading memory. They generate the same 32 characters with and without a KV cache and count 880 rows of keys and values against 43. They schedule six requests in three slots and see continuous batching finish in 9 steps where static batching takes 14. They decode speculatively with a small draft model, 13 passes for 32 characters with the same text, and quantise the model to int8, 12108 bytes down to 3543 with the same next character at every place checked. It suits a second course on machine learning or on systems performance; check your own awarding body's unit for the exact wording, none is quoted here.

Run it in your browser

The lab starts one service, jupyter, which writes the model, a training script, an int8 helper, a timing script, a starter notebook and the short scripts into its work folder, then trains the model, with nothing downloaded. JupyterLab asks for no token, because the platform's own door already knows who is opening it. The terminal beside the lab has no Python of its own, so it reaches the lab's with docker exec jupyter, followed by the command to run inside.

  1. Open the Container page. Open Build on the bar and choose Run a container. Anybody may open it, and the Git lab and the SQL lab start with no account. Every other lab needs a free account: create one, sign in, then press Start.
  2. Find the lab. The Sample labs card lists every sample. Find LLM inference: a tiny model on a CPU.
  3. Press Start. Start runs it as it is, in one press. Edit the file first opens the compose file so you can change it before you start.
  4. Open it. On the lab's page, the lab bar has a link called JupyterLab: press it to open JupyterLab in your browser and read tiny.py in the file editor. Then press Open terminal and type docker exec jupyter python3 steps/cache.py to see the same text made with and without a KV cache, and the work each way.

What to ask students to hand in

A notebook of their own that uses tiny.py to answer one question they chose, such as how the work of one decode step grows with the length of the text, how much the next character changes when the weights are quantised with one scale instead of one per row, or how the acceptance of speculative decoding changes with the number of guesses, downloaded from JupyterLab with File, Download. Then a sentence or two in their own words on what they found. Nothing is marked automatically: the sentence shows whether it was understood.

A basic lesson on this lab

LLM Inference Basics is a small lesson worked in this lab. In it a student reads the training log, watches prefill and then decode, compares a KV cache with none, schedules requests both ways, decodes speculatively and quantises the model to int8. The module How language model inference works, on a CPU, is worked in the same lab, with six lessons and six marked assignments.

Free personal accounts have it in the Basics module, which is on Home, and every college can copy it from the Library's Basics folder. It is deliberately small: it shows a teacher what a lesson worked in a lab is, and the lab does the rest. Copy it into a module of your own to change it.

The code on GitHub

The compose file has no Dockerfile: it runs one stock public image, the Jupyter Docker Stacks' own scipy-notebook, pinned to a dated tag, as its own unprivileged account with 512 MB of memory, and the model uses only NumPy. The image is licensed BSD-3-Clause and is not distributed by us. The file writes every Python file from fixed text when it starts and trains from a fixed seed, so every copy prints the same and nothing is downloaded. The limits are honest ones: the model is tiny and its text is only plausible for a few words, the real timings in the notebook change from run to run, and nothing is kept when the lab stops, so a student downloads their notebook first.

Set it as work

Say in the task what to hand in.

  1. Make the lab. In your module add a task and choose the Container lab card, then In a GitHub repository. Give https://github.com/mitsurutech/lab-inference-lab with the branch main. The compose file is the default, compose.yaml. Fork the repository first if you want to change it, for example to train the model on a text of your own.
  2. Publish and try it. Publish pins the commit you were shown, so a later change in the repository does not alter a lab you have already set. Start your own copy first.
  3. Hand it out. Assign it like any task, with a due date if you want one, and say in the task what to do and what to hand in.
  4. Mark what is handed in. Open a student’s copy and the work they handed in, mark it, and send it back with a reason if it needs more.

Try this lab, and what it costs

A free personal account can work through the basic lessons in this lab, with enough free credit each month to follow most lessons and no card. For a class, a school or college has its first 365 days free for up to 50 students, then pays per student a year, with every lab included and staff free.

Questions people ask

Why is there no token or password to type?
A lab's web page is reached only through the platform's own door, which already knows who you are and lets in only you and the staff who teach your module. So JupyterLab is started with no token. Do not run it like that on a machine anybody else can reach.
Why does the terminal use docker exec jupyter?
The lab's terminal carries a few clients and no Python. docker exec jupyter runs a command inside the lab's own container, which has Python. JupyterLab also has a terminal of its own, under File, New, Terminal.
Is this a real language model?
It is a real one, made very small: it reads characters, has one attention layer and about 4,400 numbers, and was trained on one paragraph. Its text is only plausible for a few words. Every idea it shows, the cache, batching, speculative decoding and int8, works the same way in a large model; only the sizes change.
Why do the scripts count work instead of timing it?
A real clock gives a different number on every run and every machine, so a lesson could not say what a student will see. The scripts count multiply-adds and bytes read, which are the same every time. The notebook has one cell that times the model for real, warmed up and repeated, for comparison.
Does it need a GPU?
No. The model is small enough for any CPU, and the lab is a JupyterLab with NumPy. A real serving engine on a GPU does the same things at a far larger size; the lab models the ideas and the lessons say what it leaves out.