Prefill and decode

A decoder-only autoregressive language model performs two visibly different kinds of work during generation. It first processes the supplied tokens to establish context, then generates new tokens one at a time. Those phases expose different amounts of parallelism and data reuse. Encoder–decoder architectures also run a separate encoder pass.

That distinction is durable. A fixed claim that one phase is always compute-bound and the other is always memory-bound is not.

What you will understand by the end

• What prefill and decode do, and how they contribute to user-visible latency.
• How parameter count and numeric format determine raw weight storage.
• Why long or efficiently batched prefill is often compute-sensitive, while low-batch dense decode is often bandwidth-sensitive.
• Where the first-order sustained usable bandwidth ÷ active bytes moved ceiling applies—and where it breaks.
• Why batching and quantization offer conditional gains that must be measured against latency and quality.

Core concept: two phases, different reuse patterns

Prefill processes the prompt tokens and produces the initial KV state. The model can process many prompt tokens in parallel, creating opportunities for efficient matrix operations and higher arithmetic intensity.

Decode generates the remaining output tokens autoregressively. Token N+1 depends on token N, so under standard decoding one request advances one token per iteration. A serving engine can still batch that iteration across many independent requests. Speculative decoding can instead verify several proposed tokens in one target-model invocation and accept more than one when the proposals are correct.

Phase	Common regime	What can change it
Prefill	Often compute-sensitive for long prompts or efficiently batched tokens	Short prompts, small batches, kernels, precision, memory traffic
Decode	Often bandwidth-sensitive for low-batch dense models	Batch size, context length, KV traffic, MoE routing, kernels, communication

A roofline analysis of LLM inference explains the boundary through arithmetic intensity: useful operations performed per byte moved. For example, short prefill can remain memory-sensitive and cross toward compute pressure as sequence length and reuse increase. The deployed regime depends on the model, workload, hardware, and implementation.

A model is weights, state, and operations

A model contains layers of learned parameters, or weights. Processing tokens through those layers requires matrix operations plus attention and other kernels. During low-batch decode of a dense model, each token step commonly reads most active weight bytes from device memory.

That sentence has boundaries:

• batching reuses loaded weights across several requests;
• a mixture-of-experts model activates only selected experts;
• a sharded deployment gives each GPU only its local partition;
• cache behavior, fused kernels, and offload change observed traffic.

So “a 500-token answer reads a 54 GB model 500 times” is a useful single-stream, dense-model approximation, not a general traffic measurement.

Parameters to raw weight storage

A parameter count becomes a byte count only after choosing a numeric format:

Weight format	Nominal bytes per parameter	Raw storage for P billion parameters
FP16 / BF16	2	about 2P GB
FP8 / INT8	1	about P GB
INT4 / FP4	0.5	about 0.5P GB

These are raw weight bytes, not a deployment memory budget. Quantization scales and metadata, runtime workspaces, allocator reservations, and KV state add memory. A 27B model therefore has about 54 GB of FP16 weight values, but a 54 GB device is not automatically large enough to serve it.

The narrow decode ceiling

A roofline model compares arithmetic throughput with memory bandwidth. In a batch-one, dense, weight-streaming-dominated decode regime, this estimate is useful:

ideal hardware roof
  ≈ sustained usable HBM bandwidth ÷ active bytes moved per token step

For a hypothetical 54 GB active weight set, substituting 3,350 GB/s of advertised peak bandwidth gives an absolute spec-sheet roof of about 62 token steps per second. Sustained bandwidth from a representative memory benchmark gives a more realistic upper bound. Workload-measured achieved bandwidth describes the observed operating point; it is not an independent prediction. Real execution also moves KV and activation data, runs attention and sampling kernels, and incurs launch overhead.

In a genuinely weight-bandwidth-limited batch-one regime, additional peak FLOPS alone does not raise that ideal ceiling. But that conditional statement does not mean FLOPS never matter to decode: batching, long-context attention, MoE, or different kernels can change the limiting work.

Batching trades latency for reuse

Batching lets one weight load support work for several requests. For weight-dominated dense linear layers, arithmetic intensity can rise approximately with the effective batch size, increasing aggregate throughput.

It is not free:

• each resident request consumes KV capacity;
• a larger batch can increase inter-token latency;
• new prefills can interfere with ongoing decode work;
• queueing policy changes time to first token;
• kernel efficiency and communication change with batch shape.

KV and attention traffic do not follow the simple “intensity equals batch size” model. The useful batch is therefore constrained by the latency SLO and traffic distribution, not only by a theoretical roofline crossover. Sarathi-Serve is one example of a scheduler designed specifically around this throughput–latency and prefill–decode interference problem.

Continuous batching updates the active batch between token iterations as requests arrive and finish. It improves scheduler opportunities; it does not guarantee that upstream admission, model kernels, or hardware will deliver a particular gain.

KV capacity and PagedAttention

Each active request needs KV state for its resident tokens. Long contexts or many concurrent requests can therefore make capacity the batching limit—but not invariably. An SLO, bandwidth, compute, interconnect, or scheduling overhead may bind first.

PagedAttention reduces KV-memory waste from fragmentation and supports flexible allocation and sharing. It can raise the usable batch size when KV allocation is the problem. It does not add physical memory or guarantee that capacity becomes the first limit.

Quantization changes bytes, not laws

Lower weight precision can reduce raw storage and weight traffic. That may improve model fit, latency, throughput, or cost—but byte reduction is not an automatic speedup ratio. Realized gains depend on supported kernels, dequantization and conversion overhead, batch size, KV demand, GPU architecture, and whether weights were the limiting traffic.

For example, SmoothQuant reported up to 2× memory reduction and up to 1.56× speedup for its evaluated W8A8 configurations—not an automatic 2× performance gain. Halving weight storage also does not automatically double the KV budget; the freed fraction depends on how total device memory was already divided.

Measure the latency the user experiences

The phases help decompose latency, but service metrics include more than model execution:

Metric	Main model-execution contributor	Other contributors
Time to first token (TTFT)	Prefill	queueing, scheduling, prefix-cache hits, network and runtime overhead
Inter-token latency (ITL/TPOT)	Decode iterations	batch composition, prefill interference, scheduling
End-to-end latency	TTFT plus generated-token steps	output length, queueing, network and post-processing

Measure all three. A good total latency can conceal a poor first-token experience, and a good single-stream token rate says little about throughput under a latency SLO.

Common mistakes

• Turning a common regime into a law. Prefill and decode can move between resource limits as workload shape changes.
• Treating peak bandwidth divided by model size as measured speed. It is a narrow, optimistic ceiling.
• Calling batching free. Aggregate throughput, TTFT, ITL, queueing, and KV capacity move together.
• Assuming capacity must bind first. Measure the runtime and the SLO.
• Predicting quantization gains from bit width alone. Kernel support and the original bottleneck determine realized performance; evaluation determines quality.

Practical guidance

1. Characterize prompt length, output length, concurrency, batch policy, numeric format, architecture, serving runtime, and latency targets.
2. Measure TTFT, ITL, end-to-end latency, and throughput separately.
3. Treat long-prefill → compute pressure and low-batch dense-decode → bandwidth pressure as initial hypotheses.
4. Check KV capacity, DRAM activity, tensor utilization, scheduling, and communication before attributing the limit.
5. Change one workload or resource dimension and verify that performance moves as predicted.
6. Benchmark batching and quantization for both system performance and task quality.

Summary

• Prefill and decode differ fundamentally in parallelism and data reuse; neither has one universal hardware bottleneck.
• Parameter count × bytes per parameter estimates raw weight storage, not total deployment memory.
• sustained usable bandwidth ÷ active bytes moved is an optimistic ceiling for low-batch dense decode, not a general token-rate formula.
• Batching can reuse weights and raise aggregate throughput while worsening latency or consuming KV capacity.
• Quantization reduces bytes reliably; its speed, capacity, cost, and quality effects must be measured on the actual configuration.

Knowledge check

A GPU offers 50% more peak FLOPS but the same memory bandwidth. Will it speed up a chatbot's token stream?

There is not enough information. If the measured path is genuinely weight-bandwidth-limited batch-one dense decode, extra peak FLOPS alone should not move its ideal ceiling. But batch size, context length, architecture, kernels, KV traffic, and achieved utilization can put the deployed workload in a different regime. Measure ITL and the relevant counters.

Why can batching improve throughput without being “free”?

It can reuse weights across requests and raise useful work per byte moved. At the same time, it consumes KV capacity, changes kernel efficiency, can delay individual decode iterations, and introduces queueing and prefill interference. Choose the operating point under a latency SLO rather than maximizing batch size in isolation.

An INT8 conversion halves raw weight bytes. What outcomes may you claim before benchmarking?

You may claim that INT8 nominally halves the bytes used for weight values relative to FP16. Actual encoded storage must also count scales, zero points, padding, and other format metadata. You cannot yet claim 2× speed, 2× KV capacity, half the cost, or unchanged quality. Those depend on the complete memory budget, kernels, workload, hardware, runtime, and evaluation results.

Primary sources

• LLM Inference Unveiled: Survey and Roofline Model Insights — arithmetic intensity and phase-dependent regimes.
• Sarathi-Serve — the throughput–latency tradeoff and prefill–decode interference.
• Efficient Memory Management for Large Language Model Serving with PagedAttention — KV fragmentation, allocation, and batching.
• SmoothQuant — measured memory, speed, and quality effects for W8A8 quantization.

Related evidence

• Continuous batching measured under one workload →
• Map the bottleneck hypotheses onto GPU specifications →