Easier to read with cream paper, a tint, more contrast or larger text? Choose your colours and text, once, for every page.

Free lesson

How speculative decoding works

After this lesson: I can decide which draft tokens a target model keeps in speculative decoding.

60 min lesson2 commandsLLM inference lab: a tiny language model on a CPU, with a KV cache, batching and int8

Let a small model guess first

Speculative decoding makes a language model faster with a small draft model that guesses the next few tokens. The big model then checks all the guesses in one pass.

Checking several tokens at once costs about as much as making one, because decode is memory-bound. So every good guess is a token almost for free.

By the end of this lesson

Note

I can decide which draft tokens a target model keeps in speculative decoding.

Key words

WordWhat it means
Draft modela small, fast model that guesses tokens
Target modelthe big model whose answer counts
Verifycheck several guesses in one target pass
Acceptkeep a guessed token
Acceptance ratethe share of guesses kept

The cells students fill in are left blank in this preview.

A cheap guess, then one expensive check

The draft model is tiny, so each of its guesses costs almost nothing. It guesses four tokens in a row. The target model then reads those four in one pass, the way prefill reads a prompt.

That one pass gives the target's own choice after each guess. Where the two agree, the guess is kept. A pass that reads four tokens costs about the same as one decode pass.

Compare the draft with the target

Command
docker exec jupyter python3 steps/draft.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

What the Run printed

LineWhat it shows
draft model: a table ... 361 numbersWhich character follows which, counted from the text
target model: the tiny transformer, 4355 numbersAbout 12 times bigger
the draft guesses the target's choice 38 timesOut of the first 64 positions

The cells students fill in are left blank in this preview.

Keep guesses up to the first disagreement

With greedy decoding the rule is simple. Compare each guess with the target's own choice at that place. Keep guesses while they agree.

At the first disagreement, throw that guess and every later one away, and take the target's own token instead. If all four agree, the target's pass also gives a fifth token free. Either way, every pass adds at least one token.

Try it: which guesses are kept?

Code: python
# The draft guessed four characters. The target checked them in one pass.
draft = "athe"
target = "at e."  # the target's own choice before each guess, and one more
kept = 0
while kept < len(draft) and draft[kept] == target[kept]:
    kept = kept + 1
print("guesses kept:", kept)
print("the target adds:", repr(target[kept]))
# Change target to "athe " so all four agree. What does the target add?

Run speculative decoding

Command
docker exec jupyter python3 steps/speculate.py

Shown here, not run. Press Start lab to type it into your own lab’s terminal.

Worked example: rounds 1, 2 and 5

Draft guessedKeptTarget added
"the "0"s"
"athe"2" "
" the"4" "
All 13 rounds21 of 5232 characters in 13 passes

The cells students fill in are left blank in this preview.

A common mistake

Warning

Many people think speculative decoding trades quality for speed. Every kept token is one the target itself would have chosen, so the text is the same. A poor draft only wastes time: its guesses are thrown away and the target adds one token a pass.

With sampling, keep a guess by chance

When the target samples instead of choosing the top token, matching is not enough. Say the target gives the guessed token a chance p and the draft gave it q. Keep it with chance p divided by q, or always if p is bigger.

If it is thrown away, draw from what is left instead. That is each token's p minus q where it is above zero, scaled to add up to 1. This keeps the target's own chances exactly.

Worked example: three tokens

TokenTarget pDraft qChance to keepWhat is left
a0.50.70.7140
b0.30.210.5
c0.20.110.5

The cells students fill in are left blank in this preview.

Try it: the chance to keep a guess

Code: python
# The draft gives a token chance q; the target gives it chance p.
p = 0.5
q = 0.7
if p >= q:
    keep = 1.0
else:
    keep = p / q
print("chance of keeping the guess:", round(keep, 3))
# Try p = 0.3 and q = 0.2. Why is the guess always kept?

About this lab and its source: LLM inference lab: a tiny language model on a CPU, with a KV cache, batching and int8

Teaching a class? Set this module for your students with a teacher guide, marked assignments and a lab for every student.