Speculative decoding: a small model drafts, the big model checks

The idea: the big model is slow per pass, not per token

A large language model writes one token per forward pass. Each pass has to read all of the model's weights from memory, billions of numbers, and that reading, not the arithmetic, is what takes the time. So a pass that handles 5 tokens at once costs about the same as a pass for 1 token. Speculative decoding uses this. A small, fast draft model guesses the next k tokens. The big target model then checks all k guesses in a single pass, and every guess it agrees with is a token that cost no pass of its own.

The page uses greedy decoding for both models, so you can follow every token. The cost unit is one target pass; a draft call costs 0.1 of that. Draft models are typically 10 to 50 times smaller than the target, for example a 1B model drafting for a 70B model of the same family.

One round: draft, verify, compare, append

1. Draft. The draft model writes k tokens the normal way, one call per token, each call fed its own previous guess.
2. Verify. The target runs one forward pass over the text plus the k drafts. Because a transformer computes an output at every position, it gives its own next token at all k + 1 positions: after the text, after draft 1, after draft 2, and so on.
3. Compare. Walk left to right. Keep every draft that equals the target's token at that position. At the first disagreement, drop that draft and everything after it, and take the target's own token instead: the correction. If all k drafts match, also take the target's token after the last draft: the bonus.
4. Append the kept tokens and start the next round from there.

Note the last draft “for” in that round: it equals the target's token, but it is still thrown away. It was written after “famous”, and the target's “for” was computed after “famous” too, so once “famous” is replaced neither of them means anything. Every round adds at least one token (the correction), so speculative decoding is never slower in tokens per pass than plain decoding; it can only be slower in time, because of the drafts. (Demo: one round; Demo: all drafts accepted)

Why the text does not change

With greedy decoding the argument is short: every kept token is a token the target itself chose at that position, given exactly the text before it. The output is the target's own greedy text, token for token; the draft only decides how many of them come out of one pass. The page checks this at the end of every run.

With sampling the rule is speculative sampling. Let q(x) be the draft's probability for its token x and p(x) the target's. Accept x with probability min(1, p(x) / q(x)). If it is rejected, draw the replacement from the leftover distribution max(0, p − q), renormalized. This is a kind of rejection sampling, and it is proved to produce exactly the target's distribution, so quality is unchanged. Temperature and top-p (see Decoding Strategies) are applied to p and q before the test.

Speedup and the acceptance rate

Let α be the chance that a draft token is accepted, given the ones before it were. The expected number of tokens per round is 1 + α + α² + … + α^k = (1 − α^k+1) / (1 − α). A round costs one target pass plus k draft calls, 1 + k·c. So

speedup = (1 − α^k+1) / ((1 − α)(1 + k·c))

The cost panel measures α as accepted / (accepted + rejected); drafts after a rejection were never really judged and are counted as thrown away. The chart draws the formula for the chosen k and puts this run on it as a dot.

demo	rounds	drafts: accepted / rejected / thrown	α	time	speedup
easy text, k = 4	4	10 / 2 / 3	0.83	4 + 1.5 = 5.5	✓ 2.36×
easy text, k = 8	3	11 / 2 / 11	0.85	3 + 2.4 = 5.4	✓ 2.41×
creative text, k = 4	9	6 / 8 / 16	0.43	9 + 3.0 = 12.0	1.17×
creative text, k = 8	9	6 / 8 / 37	0.43	9 + 5.1 = 14.1	✗ 0.99×

The easy answer is predictable: a small model knows that “the Eiffel” is followed by “Tower”, so most drafts pass. The poem is not: the small model writes “is a bright light”, the big one “hums a silver lullaby”, and almost every round adds only the correction. Then each draft call is pure overhead, and with k = 8 the run is slower than plain decoding. (Demo: creative text, k = 8) Real systems see the same thing: code, summaries and answers that copy from the prompt speed up 2–3×, free-form writing much less.

Choosing the draft and k

• The draft must use the same tokenizer as the target, so a draft token can be compared with a target token. In practice it is a small model of the same family.
• A better draft raises α but costs more per call; the best k grows with α. Servers often adapt k from the recent acceptance rate.
• The verification pass also needs the KV cache of the drafts; rejected positions are simply rolled back (see the KV cache in How a Large Language Model Works).
• Variants drop the separate draft model: Medusa and EAGLE add small draft heads to the target itself, prompt lookup copies the next tokens from the prompt, and lookahead decoding guesses with Jacobi iterations. The verify step is the same.

See also Decoding Strategies for greedy, beam search and top-p, the rules that pick each token, and How a Large Language Model Works for what one forward pass computes.

What the page leaves out

Both models decode greedily from fixed tables; real models compute probabilities with a transformer, and the page does not animate the sampling test min(1, p/q). The cost model is simple: a target pass costs the same for 1 and for 9 tokens, and a draft call costs 0.1. In reality a wider verification pass costs a little more, batching many users on one GPU makes the target less memory-bound and the gain smaller, and the KV cache bookkeeping, tree-shaped drafts (several guesses per position checked at once) and adaptive k are left out.

References

Leviathan, Kalman, Matias: Fast Inference from Transformers via Speculative Decoding (2022)

Chen et al.: Accelerating Large Language Model Decoding with Speculative Sampling (2023)

Hugging Face: Assisted Generation, a new direction toward low-latency text generation