Part 6 — Inference Fundamentals
Speculative decoding
Understand the technique that spends idle compute to buy back decode latency — its mechanism, the four main methods, and the single most important thing about it: when NOT to use it, because at high batch it turns itself off.
Speculative decoding is one of the most elegant ideas in inference — and one of the most misapplied. It can double decode speed for a single user, and it can also be pure overhead on a throughput server. The whole skill is knowing which situation you're in.
What you will understand by the end
- The core trade: spend idle compute to save memory-bandwidth trips.
- The generate → validate → accept mechanism every method shares.
- The four methods and what information each guesser uses.
- Why speculation disables itself at high batch — the decisive limitation.
Core idea: harvest the idle engine
Recall from Prefill and decode that at low-to-moderate batch, decode is deeply memory-bound: you haul all the weights through memory once per token, and the compute units sit mostly idle. Speculative decoding asks: the engine is idle anyway — what if we guess several tokens ahead, then check all the guesses in one pass?
Speculation spends idle compute to save memory-bandwidth trips. It generates several draft tokens cheaply, then validates them all in one forward pass — advancing multiple tokens for the cost of a single weight-load. It walks up the roofline using the spare compute that memory-bound decode leaves on the table.
The mechanism
Every method shares three steps:
- A speculator produces one or more draft tokens (cheap guesses).
- The target model (your real model) validates all drafts in one forward pass — does each draft match what the target would have produced?
- The target accepts the correct prefix and generates one more token itself.
The result is N+1 tokens per forward pass (N accepted drafts) — one weight-load, several tokens.
Generating a token is like solving a Sudoku — hard, serial, one weight-load each. Validating drafts is like checking finished Sudokus — the target processes them all at once, in parallel, in a single pass (compute-bound, like prefill). Speculation converts a stretch of serial decode into one parallel validation.
One consequence: acceptance is high early and drops deeper, and one rejection discards every draft after it — so short, high-confidence guesses beat long shaky ones. High temperature and domain mismatch both hurt acceptance.
The decisive limitation: batch size
This follows directly from "spend idle compute."
At high batch, the compute units are already saturated — there is no idle engine to spend on validating guesses. So speculation must be dynamically disabled at high batch. It is a low-batch, latency technique that trades throughput for latency: adopting it forces smaller batches, which lowers throughput and raises cost per token. It is the wrong tool for a throughput server.
This project's serving experiments run at concurrency — many requests batched together for throughput — which is exactly the regime where draft speculation auto-disables. It's a concrete "when not to use it" case: the workload wants throughput and saturates compute, so there's no idle engine to harvest. (What is relevant to that structured-output workload is the grammar-constrained variant below.) See the high-concurrency serving regime →
Dynamic speculation: match the guess to the load
Modern engines don't treat this as a hard on/off switch. Dynamic speculative decoding tunes how many draft tokens to propose from the current batch load and the measured acceptance rate: long, aggressive drafts when the engine is idle, tapering to short drafts or none as the batch fills and compute saturates. The effect is that speculation self-regulates — it harvests idle compute when there is some and quietly steps aside when there isn't — so it can be left enabled without punishing the throughput case. The principle is unchanged ("spend idle compute"); the engine just measures how much idle compute exists, moment to moment, instead of relying on a fixed draft length.
The four methods — who guesses, from what information
| Method | Who guesses | Information used | Best at |
|---|---|---|---|
| Draft-target | a separate small model | the output words so far | quick setup, no training |
| EAGLE | a tiny head on the target | the target's hidden states (its internal "thoughts") | general use, highest accuracy |
| N-gram | no model — a dictionary of the input | repeated patterns in the input | code, repetitive output |
| Grammar-constrained | the schema/state machine | the rules of valid structure | JSON, tool calls, SQL |
The progression is "how good is the guesser's information?" — output words → internal thoughts → literal input repeats → hard structural rules. Better information means higher acceptance, which means fewer memory trips.
- Draft-target runs a second small model; simple, no training, most overhead.
- EAGLE trains a small head that reads the target's hidden states, so its guesses are far more accurate — the go-to general method when you can train the head.
- N-gram needs no model at all: it predicts from repeats in the input, which is why it dominates code (the output largely mirrors the input) — often 10+ accepted tokens.
- Grammar-constrained is a different flavor: it constrains output to a schema (JSON, tool call) with a state machine that masks illegal tokens. When only one token is legal it's forced — trivially predictable — so structured output flies through for free.
Where it works — and where it doesn't
The recipe for a real win is low/moderate batch (idle compute) + decode-heavy (long outputs) + predictable output (high acceptance). Single-user chat, voice agents, long chain-of-thought reasoning, and code completion all fit — often 2–3× tokens/sec at batch 1. A high-batch throughput server fits none of them.
Common mistakes
- Turning it on for a throughput server. High batch → it auto-disables, and forcing it shrinks the batch and hurts throughput.
- Expecting a prefill/TTFT improvement. It speeds decode (tokens/sec) only; it does nothing for time-to-first-token.
- Long draft sequences at high temperature. Acceptance drops deep and one miss voids the rest — favor short, confident drafts.
- Ignoring domain fit. A draft model good at one domain guesses poorly in another; n-gram shines only when output repeats the input.
Practical guidance
- Turn it on for latency-critical, low-batch, decode-heavy work (voice, single-user chat, reasoning); leave it off for high-batch throughput.
- Pick by domain: n-gram for code/structured repetition, EAGLE for general low-batch latency, draft-target when you can't train a head, grammar-constrained for JSON/tool/SQL output.
- If you serve structured output at batch, the relevant relative is grammar constraint, not draft speculation.
- Measure acceptance rate — it's the lever that decides whether the win is real.
Summary
- Speculation spends idle compute to save memory trips via generate → validate → accept, yielding N+1 tokens per pass.
- Validation is cheap because it's parallel (like prefill), not serial (like decode).
- It auto-disables at high batch — a low-batch, latency technique, wrong for throughput servers.
- Choose the method by the guesser's information: draft-target, EAGLE (hidden states), n-gram (input repeats), grammar (structure).
Knowledge check
A voice agent (one user, latency-critical) and a batch-throughput image server both ask "should we enable speculative decoding?" Answer each and say why the same technique flips.
Voice agent: yes — low batch means idle compute to harvest, it's decode-heavy, and latency is what the user feels. Throughput server: no — high batch saturates compute so speculation auto-disables, and forcing it would shrink the batch and cut throughput. The technique didn't change; the workload's position on the throughput-vs-latency dial did.
Which GPU resource does speculation spend, and which does it save?
It spends compute (the idle engine, used to validate drafts in parallel) to save memory-bandwidth trips (weight-loads). That's why it only works when compute is idle — i.e. at low batch.