Speculative decoding: proposals, verification, and acceptance

Ordinary autoregressive decoding advances one accepted token per serial target-model iteration. Speculative decoding tries to advance farther: propose several future tokens cheaply, score them together with the target, accept a valid prefix, and correct the first rejection.

It is not “free compute at batch one,” and batch size is not a universal on/off boundary. A configuration wins only when accepted progress repays proposal, verification, memory, state-management, and scheduling overhead for the actual deployment.

What you will understand by the end

• How greedy verification differs from exact speculative sampling.
• Why K + 1 tokens is a maximum, not a guarantee.
• The main proposal families: draft models, EAGLE/Medusa/MTP, self-speculation, n-gram lookup, and lookahead-style methods.
• Why high batch is a cost warning rather than a universal rejection rule.
• Which measurements establish a real latency or throughput win.

A linear speculative cycle

For a chain of proposals:

1. A proposer generates K draft tokens.
2. The target scores the proposed positions in one verification iteration.
3. The runtime accepts a valid prefix.
4. It emits a target or correction token at the first rejection boundary.
5. Rejected speculative state is discarded or rolled back.

Draft proposes:       A  B  C  D
Target verification: ✓  ✓  ✗
Later draft state:             discarded
Correction token:          X

Cycle advances: A B X

If w drafts survive before the first rejection, a common linear cycle advances by w + 1 tokens. If all K survive, the maximum is K + 1. Expected progress depends on acceptance by depth—not merely the maximum draft length.

Greedy matching versus exact sampling

Under greedy decoding, a draft token survives while it matches the target’s argmax sequence. That simple token-matching explanation does not cover stochastic sampling, where there is no single predetermined “target token.”

Under classical exact speculative sampling, acceptance uses the draft distribution q and target distribution p. Modified rejection sampling accepts or rejects proposals probabilistically and draws the correction from a residual distribution so the final sequence follows the target distribution within normal numerical limits.

In a linear chain, the first rejection invalidates later drafts because they depended on the rejected prefix. A tree proposes several branches to increase the chance of a long accepted path, but verifies more proposed tokens. Tree width and depth must repay their verification and memory cost.

The performance equation

The useful comparison is progress per unit time:

baseline progress rate
= 1 accepted token / target decode-step latency

speculative progress rate
= expected output tokens advanced per cycle
  / (proposal + target verification
     + acceptance/state + scheduling time)

Speculation wins when the second rate is higher under the same quality, latency, and capacity requirements. Batch matters because it changes verification shape and hardware utilization, but it is not the equation.

In low-batch, weight-traffic-dominated decode, verification may use spare arithmetic capacity and reuse target weights over more token positions. In long-context regimes, reducing serial KV scans or using a sparse-KV drafter may matter. Very cheap auxiliary heads, lookup proposals, tree selection, and batch-aware draft lengths create still other cost profiles.

High batch: warning, not law

Conventional fixed-length speculation often loses effectiveness as batch grows and target verification becomes expensive. It may remain enabled and regress; a runtime may require manual tuning; or a policy may shorten or disable drafting based on active batch or observed acceptance. Never assume automatic protection.

High-throughput workloads are not categorically excluded. MagicDec reported gains in evaluated long-context workloads at batches from 32 through 256, where KV traffic changes the economics. Lightweight target-conditioned heads and batch-aware systems can also produce throughput gains.

Regime	Common expectation	Important exceptions
Low batch, short/moderate context	Often favorable if the proposal is cheap	Small target or poor acceptance can regress
High batch, short context	Conventional verification often becomes costly	Lightweight heads and batch-aware policies may help
Long context	KV traffic can make speculation useful again	Drafter KV design and acceptance remain critical
Repetitive/editing workload	Lookup drafting can be attractive	Literal reusable overlap must exist
High-entropy sampling	Agreement often falls	Strong target-conditioned drafters may remain viable
Guided structured output	Constraints may improve agreement	Guidance remains separate and adds integration work

Speculation may add a draft checkpoint, KV state, verification tokens, temporary buffers, or scheduler complexity and reduce feasible concurrency. Other methods use small heads, the target itself, or CPU lookup and may have a different memory footprint. Measure the selected implementation rather than assuming every method forces smaller batches.

Proposal families

Proposal family	How drafts are produced	Examples	Main cost or risk
Separate draft model	Smaller autoregressive model	Classical draft–target	Extra weights/KV, serial draft steps, tokenizer/model fit
Target-conditioned predictor	Auxiliary heads or lightweight feature predictor	Medusa, EAGLE, MTP, recurrent drafter	Compatible trained modules, target integration, tree/sequence verification
Self-speculation	Cheaper approximation of the target	Early exit, layer skipping, sparse-KV drafting	Approximation design, target modifications, acceptance
Retrieval or lookup	Reuses spans from token history	N-gram, prompt lookup	Works only when useful literal repeats exist
Iterative parallel proposal	Produces blocks non-autoregressively	Lookahead/Jacobi, block proposals	Convergence, proposal overhead, engine maturity

Draft–target

A smaller model proposes a chain, and the full model verifies it. Setup is conceptually simple, but the draft must be cheap, align with the target and tokenizer, and justify its extra weights, KV state, and execution path.

Medusa, EAGLE, and MTP

Medusa adds heads that predict future tokens and may form a proposal tree. EAGLE uses a lightweight target-conditioned drafter to predict future target features and tokens, then verifies proposed paths with the target. MTP uses model-native or trained multi-token prediction modules. These labels do not imply one permanent winner; support, acceptance, memory, and speed depend on the model and engine.

Use EAGLE when a compatible drafter checkpoint and engine path exist—or when fleet-scale benefit justifies training and maintaining one. Compare it with MTP, Medusa, draft–target, self-speculation, and lookup on the exact serving stack.

N-gram and prompt lookup

Lookup drafting searches prior token history—often the prompt plus generated output, and in some systems a shared pool—for a matching suffix, then proposes the tokens that followed an earlier match. It can be attractive for editing, translation, templated text, repetitive conversations, and some code workflows, but novel generation may offer little literal overlap.

Acceptance is workload-specific. A current TensorRT-LLM study reported average accepted length near 1.3 for generic first-turn chat, 1.66 for a second conversation turn, and above 4 for its repetitive translation setup. Those are named experiments, not general values.

Guided decoding is orthogonal

Grammar or guided decoding restricts the legal next-token set. By itself, it does not propose several tokens and verify them in parallel, so it is not a speculative-decoding family.

A runtime can combine guidance with draft–target, EAGLE, MTP, or another method, but must coordinate grammar masks and state for draft and target, advance speculative grammar state, and roll it back after rejection. Grammar compilation, mask generation, mask application, and CPU/GPU synchronization may add work.

Measure guided decoding alone and guidance plus speculation separately. Also verify feature combinations with continuous batching, prefix caching, LoRA, quantization, parallelism, logprobs, and the desired sampler.

What this project’s evidence does—and does not—show

Benchmark checklist

Record the baseline and speculative configurations, then sweep batch, context, output length, temperature, top-k/top-p, and offered load. Measure:

• proposal latency and target verification latency;
• draft-token acceptance rate, average accepted length, and acceptance by depth;
• proposed and verified tokens, plus target iterations per output token;
• p50/p95/p99 inter-token latency and request throughput;
• TTFT, GPU memory, and maximum SLO-compliant concurrency;
• output/distribution or task-quality equivalence;
• scheduler behavior and every combined feature required in production.

Acceptance rate alone is insufficient. A drafter can accept many tokens and still lose if proposal or verification is too expensive.

Published results range from regressions to multi-fold gains across models and setups. Cite a speedup with its model, drafter, engine, hardware, sampling, batch, and context—and reproduce it on the deployment trace.

Summary

• Speculation trades proposal and verification work for fewer serial target iterations.
• Greedy matching is not exact stochastic sampling; verify the acceptance contract.
• K + 1 tokens is the all-accepted maximum, not the expected result.
• Batch affects cost but does not universally disable or disqualify speculation.
• Proposal families include draft models, auxiliary predictors, self-speculation, lookup, and iterative parallel methods; guided decoding is separate.
• Accepted progress per total cycle time—not acceptance rate alone—decides the win.

Knowledge check

A drafter proposes five tokens and the first two pass. How far does a standard linear cycle usually advance?

Three tokens: two accepted drafts plus one target/correction token. Six is only the maximum when all five drafts survive.

Why is greedy token matching not a complete explanation of speculative sampling?

Stochastic sampling has no single predetermined target token. Exact speculative sampling uses draft and target probabilities for modified rejection sampling and draws a correction so the output follows the target distribution.

Does high batch prove speculation should be disabled?

No. It often increases conventional verification cost, but long-context, lightweight-head, sparse-KV, and batch-aware methods may still help. Benchmark the actual method, context, batch, engine, and SLO.

Is grammar-constrained decoding a speculative method?

No. It restricts legal tokens. It can be combined with speculation, but ordinary grammar masking does not itself propose and verify multiple target tokens per iteration.

Which measurements are more informative than acceptance rate alone?

Proposal latency, verification latency, accepted tokens per cycle, ITL distribution, and throughput. Also check memory, maximum SLO-compliant concurrency, and output equivalence.

Primary sources and version boundary

Technical and product claims were reviewed August 3, 2026. Speculative support is model-, tokenizer-, sampler-, engine-, hardware-, and feature-combination specific.

• Fast Inference via Speculative Decoding and Speculative Sampling
• EAGLE and Medusa
• MagicDec long-context, high-throughput evaluation
• TensorRT-LLM speculative-decoding families
• TensorRT-LLM batch-aware draft-length API
• TensorRT-LLM n-gram evaluation
• Combining guided and speculative decoding
• SGLang speculative decoding

Related chapters

• Prefill and decode
• GPU architecture for LLM inference
• Batching and concurrency
• Inference runtimes