Let a small model guess first
Speculative decoding makes a language model faster with a small draft model that guesses the next few tokens. The big model then checks all the guesses in one pass.
Checking several tokens at once costs about as much as making one, because decode is memory-bound. So every good guess is a token almost for free.
By the end of this lesson
Note
I can decide which draft tokens a target model keeps in speculative decoding.
Key words
| Word | What it means |
|---|---|
| Draft model | a small, fast model that guesses tokens |
| Target model | the big model whose answer counts |
| Verify | check several guesses in one target pass |
| Accept | keep a guessed token |
| Acceptance rate | the share of guesses kept |
The cells students fill in are left blank in this preview.
A cheap guess, then one expensive check
The draft model is tiny, so each of its guesses costs almost nothing. It guesses four tokens in a row. The target model then reads those four in one pass, the way prefill reads a prompt.
That one pass gives the target's own choice after each guess. Where the two agree, the guess is kept. A pass that reads four tokens costs about the same as one decode pass.
Compare the draft with the target
docker exec jupyter python3 steps/draft.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
What the Run printed
| Line | What it shows |
|---|---|
| draft model: a table ... 361 numbers | Which character follows which, counted from the text |
| target model: the tiny transformer, 4355 numbers | About 12 times bigger |
| the draft guesses the target's choice 38 times | Out of the first 64 positions |
The cells students fill in are left blank in this preview.
Keep guesses up to the first disagreement
With greedy decoding the rule is simple. Compare each guess with the target's own choice at that place. Keep guesses while they agree.
At the first disagreement, throw that guess and every later one away, and take the target's own token instead. If all four agree, the target's pass also gives a fifth token free. Either way, every pass adds at least one token.
Try it: which guesses are kept?
# The draft guessed four characters. The target checked them in one pass.
draft = "athe"
target = "at e." # the target's own choice before each guess, and one more
kept = 0
while kept < len(draft) and draft[kept] == target[kept]:
kept = kept + 1
print("guesses kept:", kept)
print("the target adds:", repr(target[kept]))
# Change target to "athe " so all four agree. What does the target add?
Run speculative decoding
docker exec jupyter python3 steps/speculate.pyShown here, not run. Press Start lab to type it into your own lab’s terminal.
Worked example: rounds 1, 2 and 5
| Draft guessed | Kept | Target added |
|---|---|---|
| "the " | 0 | "s" |
| "athe" | 2 | " " |
| " the" | 4 | " " |
| All 13 rounds | 21 of 52 | 32 characters in 13 passes |
The cells students fill in are left blank in this preview.
A common mistake
Warning
Many people think speculative decoding trades quality for speed. Every kept token is one the target itself would have chosen, so the text is the same. A poor draft only wastes time: its guesses are thrown away and the target adds one token a pass.
With sampling, keep a guess by chance
When the target samples instead of choosing the top token, matching is not enough. Say the target gives the guessed token a chance p and the draft gave it q. Keep it with chance p divided by q, or always if p is bigger.
If it is thrown away, draw from what is left instead. That is each token's p minus q where it is above zero, scaled to add up to 1. This keeps the target's own chances exactly.
Worked example: three tokens
| Token | Target p | Draft q | Chance to keep | What is left |
|---|---|---|---|---|
| a | 0.5 | 0.7 | 0.714 | 0 |
| b | 0.3 | 0.2 | 1 | 0.5 |
| c | 0.2 | 0.1 | 1 | 0.5 |
The cells students fill in are left blank in this preview.
Try it: the chance to keep a guess
# The draft gives a token chance q; the target gives it chance p.
p = 0.5
q = 0.7
if p >= q:
keep = 1.0
else:
keep = p / q
print("chance of keeping the guess:", round(keep, 3))
# Try p = 0.3 and q = 0.2. Why is the guess always kept?