Transformer inference mechanics: QKV, KV cache, and attention

The terms KV cache, GQA, FlashAttention, PagedAttention, and continuous batching describe changes at different layers of an inference stack. They do not all optimize one piece of Transformer math. This chapter follows one token through the model, then places each technique at the boundary it actually changes.

What you will understand by the end

• What Q, K, and V do in causal self-attention.
• What a conventional decoder stores in its per-layer KV cache.
• How MHA, MQA, and GQA trade cached state against model quality and capacity.
• How FlashAttention, PagedAttention, and continuous batching differ.
• Why decode cost includes both active model weights and KV state.

The computation in one pass

A token ID maps to a learned embedding and passes through decoder blocks. Each block usually contains causal self-attention and a position-wise feed-forward network, plus normalization, residual connections, and positional treatment such as RoPE.

Attention explicitly mixes information across permitted token positions. In a standard dense Transformer, normalization and the MLP apply the same transformation to each position independently. Position-wise does not mean inexpensive: projections and MLPs contain substantial weights and can dominate FLOPs and low-batch weight traffic.

The final LM head projects the hidden state to vocabulary logits. A decoding policy—such as greedy selection, sampling, constrained decoding, or speculative verification—then determines the next accepted token. A large vocabulary projection can itself be a meaningful execution and memory cost.

What Q, K, and V do in causal attention

Each attention layer learns projections that produce queries, keys, and values. A useful memory aid is:

• Q (query): what pattern is this position seeking?
• K (key): what pattern does a position advertise?
• V (value): what information is aggregated if that position receives weight?

These are metaphors, not fixed human-assigned meanings. A learned head need not map cleanly to syntax, reference, or any single interpretable function.

Per head, decoder attention is conceptually:

Attention(Q, K, V)
= softmax((QKᵀ / √d_k) + causal_mask) V

The causal mask prevents a position from using disallowed future positions. Implementations may fuse scaling, masking, bias, softmax, and value aggregation.

At causal position t, the query can attend only to permitted keys and values up to t. During incremental generation, the runtime processes the newly appended token to produce logits for the following token.

What the KV cache stores per layer

Without caching, every decode iteration would rerun the growing prefix and repeatedly recompute earlier layer activations and K/V projections. For standard full attention, each such full-prefix forward includes attention work quadratic in prefix length.

With a KV cache, each iteration computes projections for the new token and scans the stored K/V. The new-token attention work and traffic at a conventional full-attention layer therefore grow linearly with resident context.

Layer 1: K[1…t], V[1…t]  +  new K/V for t+1
Layer 2: K[1…t], V[1…t]  +  new K/V for t+1
  …
Layer L: K[1…t], V[1…t]  +  new K/V for t+1

For a conventional decoder, a useful logical estimate is:

KV bytes
≈ 2 × layers × resident tokens
  × KV heads × head dimension
  × bytes per cached element

The factor two represents keys and values. Multiply by active sequences when their state is not shared. Allocator blocks, padding, prefix sharing, cache quantization, sliding windows, sparse attention, and latent or hybrid state change physical allocation.

Cost	Scales mainly with	Why it matters
Model weights	Active parameters and precision	Fixed resident footprint; often major low-batch decode traffic
KV state	Layers, tokens, KV width, precision, active requests	Variable context/concurrency footprint and attention traffic
Temporary/workspace memory	Kernels, shapes, graph capture, runtime	Can constrain feasible token budgets or compilation
Communication	Parallelism and topology	Can dominate distributed execution

After weights and runtime allocations are resident, KV state is often the largest term that grows with tokens and requests. It can constrain context or concurrency, but another capacity or latency limit may bind first.

MHA versus MQA versus GQA

Multiple heads use distinct learned projections to attend through different representation subspaces. Head count and width are architecture choices; simply adding heads does not guarantee better quality.

• MHA: each query head has its own K/V head.
• MQA: all query heads share one K/V head.
• GQA: groups of query heads share K/V heads.

Cache width is KV heads × head dimension, not query-head count alone. If a model doubles its MHA head count while halving head dimension at fixed hidden width, total K/V width can remain unchanged.

MQA and GQA are model-architecture efficiency trade-offs. The MQA paper reported minor quality degradation in its experiments; GQA was introduced as a middle ground seeking quality near MHA with much of MQA’s efficiency. Measure quality on the target tasks and performance on the target runtime.

Modern blocks may also use RMSNorm, RoPE, and gated MLPs such as SwiGLU. They solve different problems—normalization, positional structure, and feed-forward transformation—and are not one category of “lighter equivalents.” RoPE applies position-dependent rotations to Q and K so dot products encode relative-position structure; its cost and fusion depend on the implementation.

FlashAttention versus PagedAttention

Both preserve the intended attention computation rather than changing the trained model architecture, though kernel order and precision can produce normal floating-point differences.

FlashAttention is an IO-aware exact attention algorithm. It tiles attention through on-chip memory, reduces HBM traffic, and avoids materializing the full score matrix. Its smaller temporary-memory footprint can permit larger prefill shapes, but it does not by itself reduce persistent KV bytes per resident token.

PagedAttention lets attention address K/V stored in non-contiguous blocks. That enables on-demand allocation, lower fragmentation, sharing, and copy-on-write behavior. It can increase usable concurrency when allocation waste or duplication is limiting, but it creates no physical memory and guarantees neither better latency nor a larger SLO-compliant batch.

Technique	Stack layer	Structural change	What it does not guarantee
KV cache	Runtime state	Reuses prior-layer K/V instead of rerunning the prefix	That KV traffic dominates decode
MQA/GQA	Model architecture	Reduces KV heads and cached state	Unchanged quality or proportional throughput
FlashAttention	Algorithm/kernel	Reduces attention IO and temporary materialization	Smaller persistent KV state
PagedAttention	KV layout + kernel	Supports non-contiguous allocation and sharing	More physical memory or better latency everywhere
Continuous batching	Scheduler	Changes the active set at iteration boundaries	End-to-end concurrency or a fixed throughput gain

ORCA and continuous batching

Earlier generative systems could use request-level or static batches whose membership remained fixed until completion. Short requests waited behind long ones, and new requests waited for the current batch.

ORCA introduced iteration-level scheduling: after a model iteration, completed requests can leave and newly admitted requests can join. It paired this with selective batching for operations that benefit from batching. This is a direct ancestor of what modern systems call continuous or iteration-level batching; current schedulers add their own admission, preemption, priority, chunked-prefill, and memory policies.

A better connection map

When you meet a new optimization, first identify its layer:

• Model architecture: MHA, MQA, GQA, MLA, dense versus MoE.
• Numerical format: weight, activation, or KV precision.
• Algorithm/kernel: FlashAttention, fusion, CUDA graphs.
• Runtime state and allocation: KV representation, paging, prefix sharing.
• Scheduling: continuous batching, chunked prefill, preemption, priorities.
• Parallelism and communication: tensor, pipeline, data, and expert parallelism.
• Admission and routing: which work reaches which replica and when.

Leading engines usually provide optimized attention, dynamic KV allocation, and iteration-level batching. Do not assume literal PagedAttention or identical behavior; verify the engine version, model, platform, precision, distributed mode, and feature combination.

Common mistakes

• Treating KV state as the whole decode bandwidth or capacity budget.
• Inferring KV bytes from query-head count without head dimension.
• Treating MQA/GQA as free performance with unchanged quality.
• Calling FlashAttention and PagedAttention interchangeable “fast attention.”
• Attributing an end-to-end admission fix to one scheduler algorithm.
• Comparing feature labels without checking the implementation and workload.

Summary

• Causal attention produces per-layer K/V state that incremental decode retains.
• Dense decode traffic includes active weights and context-dependent KV reads.
• Fewer KV heads structurally reduce KV bytes; quality and system speed remain empirical.
• FlashAttention changes attention IO; PagedAttention changes KV layout and allocation.
• Continuous batching changes scheduling, and requires concurrency through the full path.

Knowledge check

A dense model has a small KV cache but slow batch-one decode. Is that a contradiction?

No. Batch-one dense decode may be dominated by streaming model weights rather than KV state. Measure active weight bytes, KV traffic, achieved bandwidth, kernel time, and launch gaps before assigning the bottleneck.

A model changes from 32 MHA heads to 64 while keeping hidden width fixed. Must its KV cache double?

No. If head dimension halves, total K/V width can remain unchanged. Cache bytes depend on KV heads × head dimension, layers, resident tokens, precision, and active sequences.

What does GQA guarantee?

At fixed head dimension and cache precision, fewer KV heads guarantee fewer cached K/V values. GQA does not guarantee unchanged quality or a particular throughput gain.

Why is FlashAttention not simply a fast version of PagedAttention?

FlashAttention changes the IO strategy for exact attention computation. PagedAttention changes how persistent KV blocks are allocated, shared, and addressed. They solve different primary problems and can coexist.

What did the project’s roughly 22× result establish?

Correcting router and replica admission let concurrent work reach an engine that already supported continuous batching. It established an end-to-end concurrency-propagation problem for that deployment, not the standalone gain of one scheduler algorithm.

Primary sources and version boundary

Technical claims were reviewed August 2, 2026. Engine implementations change; verify the exact model, runtime, hardware, precision, and workload.

• Attention Is All You Need
• Multi-Query Attention and Grouped-Query Attention
• RoFormer / RoPE, RMSNorm, and GLU variants
• FlashAttention
• PagedAttention / vLLM
• ORCA iteration-level scheduling
• LLM inference roofline analysis
• Evaltude’s bounded batching experiment

Related chapters

• Prefill and decode
• KV cache and context growth
• Attention variants
• Batching and concurrency
• Inference runtimes