Speculative decoding
A small draft model proposes several tokens and the large model verifies them in one pass, giving faster decoding without changing the output distribution.
The bottleneck
Generating K tokens normally takes K serial runs of the large model, because each token depends on the one before it1. Each run reads all the weights from memory to produce one token, which wastes much of the GPU’s compute (Latency and cost).
The trick
Two research groups described the same idea independently. Use a cheaper draft model to guess several next tokens, then have the large target model check all the guesses in a single parallel pass12. Leviathan and colleagues rest the idea on two observations: hard text-generation tasks contain easier parts that a smaller model approximates well, and a large model can score several tokens in parallel for about the cost of one step1.
Diagram: my own.
Steps:
- The draft model proposes a short run of tokens.
- The target model computes its own probabilities for those positions in one pass.
- A modified rejection-sampling rule accepts a prefix of the proposed tokens and, at the first rejection, replaces that token with one sampled from the target’s distribution2.
- Repeat from the new position.
Every round yields at least one target-quality token, and sometimes several.
What it guarantees
Both papers stress that the output distribution is unchanged. Leviathan et al. state the method samples from the large model without changing the outputs, using no retraining or architecture change1. DeepMind’s version preserves the target model’s distribution, subject to hardware numerics, and reported a 2 to 2.5 times decoding speedup on a 70-billion-parameter Chinchilla model in a distributed setup2. That speedup is specific to their model, hardware and tasks.
The “same distribution” claim is about sampling. With greedy decoding the results should match what the target model would produce, up to numerical differences.
What decides the speedup
- Acceptance rate. If the draft often guesses what the target would say, many tokens are accepted per pass. Predictable text, such as code boilerplate, quotations or repeated structure, accepts more than creative text.
- Draft cost. The draft must be much cheaper than the target, or the gains vanish.
- Load. The method spends spare compute to save memory-bound steps. When a server is already busy with large batches, there is less spare compute, so the benefit shrinks (my understanding; test it at your own concurrency).
- Extra memory. The draft model needs space too.
Variants
Engines now offer several ways to get drafts. vLLM’s repository documentation lists speculative decoding with options such as n-gram lookup, suffix methods and EAGLE-style draft heads3. Check the engine’s current docs, since this is a fast-moving area.
When to use it
- You self-host a large model for interactive use, and decode speed is the problem.
- Outputs are long and fairly predictable.
- You can measure acceptance rate and latency on real traffic.
It does not help prefill, so long prompts with short answers see little gain. If you use a hosted API, any such acceleration is the provider’s concern, though you may see it described as a faster output mode. Combine it with other levers: quantisation shrinks the weights that each step reads, and batching raises throughput.
Sources
- [1]Fast Inference from Transformers via Speculative Decoding — Leviathan, Kalman, Matias · 2023PARAPHRASED
- [2]Accelerating Large Language Model Decoding with Speculative Sampling — Chen et al., DeepMind · 2023PARAPHRASED
- [3]vLLM: High-Throughput LLM Inference and Serving Engine (repository overview) — vLLM project, via docsearchFEATURE LIST, AS SUMMARISED BY A THIRD PARTY