MHA, MQA, GQA and MLA: KV-cache tradeoffs

Attention variants change what state an autoregressive decoder retains for earlier tokens. For conventional multi-head attention (MHA), multi-query attention (MQA), and grouped-query attention (GQA), the number of key/value heads directly changes the attention portion of the KV cache. Multi-head latent attention (MLA) uses a different compressed-state design and needs its own accounting.

The architectural savings are real. Their effect on end-to-end throughput, latency, cost, and quality is conditional on the checkpoint, workload, context lengths, runtime, kernels, parallelism, and hardware.

What you will understand by the end

• How num_key_value_heads changes conventional attention KV-cache size.
• What MHA, MQA, and GQA share—and what lowering the K/V-head count guarantees.
• Why reduced K/V state can improve serving without guaranteeing faster end-to-end decode.
• How MLA's compressed latent and positional state differ from GQA-style head sharing.
• What model configuration and benchmark evidence you need before making a serving choice.

How num_key_value_heads changes KV-cache size

For a conventional decoder whose layers use the same K/V-head shape, the attention cache is:

KV bytes = 2 × layers × batch × cached tokens × kv_heads × head_dim × bytes_per_element
           └ K and V

Holding layer count, batch, cached sequence length, K/V head dimension, and precision fixed, the attention KV-cache component scales linearly with kv_heads. Halving the K/V-head count halves that component. This ratio does not automatically cover sliding-window truncation, quantized cache layouts, architecture-specific state, allocator overhead, or other GPU memory such as weights and workspaces.

Past query vectors are not retained in the autoregressive KV cache. Query heads are not free, however: the current token still needs query projections, query–key scores, softmax, value aggregation, output projection, and efficient kernel execution.

MHA vs MQA vs GQA

For an eight-query-head illustration:

  MHA — one K/V head per query head                 8 K/V heads
    Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
    K1 K2 K3 K4 K5 K6 K7 K8
    V1 V2 V3 V4 V5 V6 V7 V8

  GQA — one K/V head per query group               2 K/V heads
    [ Q1 Q2 Q3 Q4 ]      [ Q5 Q6 Q7 Q8 ]
          K1                    K2
          V1                    V2

  MQA — one K/V head shared by every query head    1 K/V head
    Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
              K  (shared)
              V  (shared)

The conventional attention KV-cache ratios are 1, 2/8, and 1/8 in this example. Those ratios assume equal K/V head dimension and precision, the same layers, sequences, and batch, and no architecture-specific extra cached state.

• MHA — multi-head attention. Each query head has its own K and V head. This is the baseline design from the original Transformer—not a guarantee that a whole model will have better quality than a different model using another variant.
• MQA — multi-query attention. Every query head shares one K/V head. This gives the smallest conventional K/V-head count and can reduce incremental-decoding traffic substantially. Shazeer's controlled experiments reported much faster decoding with minor quality degradation; that result belongs to those models and evaluations.
• GQA — grouped-query attention. Query heads are divided into groups, with one K/V head per group. In controlled uptraining experiments, Ainslie and colleagues found GQA quality close to MHA with speed comparable to MQA. This is empirical evidence for that setup, not a universal quality ordering among independently trained checkpoints.

Sharing K/V heads reduces representational degrees of freedom relative to an otherwise comparable MHA layer, so quality must be evaluated. It does not follow that every MHA model outperforms every GQA or MQA model: data, scale, training, tokenizer, objective, and post-training can dominate cross-model comparisons.

Guaranteed effect versus conditional serving effect

Variant	Cached attention state	Guaranteed architectural effect under equal dimensions	Conditional serving effect
MHA	Per-head K and V	Largest conventional K/V-head count in a controlled comparison	Baseline; may consume more KV capacity and traffic
GQA	One K/V head per query group	K/V component scales with group count	Often an effective efficiency/quality compromise; benchmark it
MQA	One shared K/V head	Minimum conventional K/V-head state	Can improve decode throughput; quality and runtime dependent
MLA	Compressed K/V latent plus positional key state	Cache follows latent and positional dimensions, not a K/V-head ratio	Potentially large savings; kernel, phase, and hardware dependent

Fewer conventional K/V heads reduce attention-cache bytes stored and read. That often helps most at long context or high concurrency, when KV traffic or capacity is material. It does not prove end-to-end decode will speed up. Model-weight reads, FFN or MoE compute, attention kernels, occupancy, tensor-parallel communication, launch or scheduler overhead, quantization work, and short contexts can bind first.

Architecture-level KV savings can also interact with tensor-parallel sharding or replication and specialized kernel support. Validate per-GPU memory and throughput on the intended runtime rather than inferring them from the global cache formula alone.

How MLA differs from GQA

Multi-head latent attention, introduced in DeepSeek-V2, is not another integer setting for kv_heads. It trains the model around a learned low-dimensional K/V bottleneck. In the DeepSeek design, each token retains a compressed K/V content latent plus a separate positional key component used with decoupled rotary position encoding. Its cache therefore depends on latent and positional dimensions, precision, and implementation—not the MHA/GQA head-count ratio above.

Efficient execution has more than one algebraically valid path. Current vLLM documentation describes a compute-friendly, MHA-like path and a data-movement-friendly, MQA-like path. Through matrix absorption, the decode-oriented path can operate over the compressed latent without materializing full per-head K/V tensors for the entire cached context. Which path is appropriate depends on phase, shapes, kernels, and hardware.

Capacity benefit and traffic benefit are different claims

A smaller attention cache can provide:

• Capacity headroom: longer contexts or more concurrent sequences when KV memory is the binding resource. Weights, workspaces, CUDA graphs, fragmentation, runtime buffers, sequence and token caps, latency SLOs, or external demand may bind first.
• Less K/V traffic: fewer bytes for the attention kernel to read during decode. This can improve throughput when that traffic is material and the runtime has efficient kernels.

These claims need different evidence. Cache occupancy measures capacity use; it does not measure bandwidth. A cache can occupy little memory and still be read repeatedly enough to consume meaningful bandwidth.

Mental model

Past query vectors are not cached, but query-head computation is not free. In conventional MHA, MQA, and GQA, fewer K/V heads linearly reduce the attention KV-cache component when head dimension and precision are fixed. That can improve capacity and reduce decode traffic, especially at long context or high concurrency, but end-to-end speed depends on the workload, model, hardware, parallelism, and kernels. MLA caches a learned compressed K/V latent plus positional state and requires separate implementation-aware accounting. Quality and serving gains must be measured rather than inferred from the attention label.

Common mistakes

• Calling query heads free. Their past vectors are not cached; their current-token projection and attention work still execute.
• Turning a byte reduction into a speed guarantee. Fewer K/V heads reduce one traffic component. Profile the whole model and runtime before naming the bottleneck.
• Assuming a monotonic quality ranking. Controlled architecture comparisons can expose a trade-off; unrelated checkpoints cannot be ranked by attention label alone.
• Putting MLA on the K/V-head ratio line. Its latent and positional dimensions require a different cache formula and specialized execution paths.
• Calling GQA a universal modern default. It is common, not exclusive. Read the config.
• Expecting to enable an architecture at serve time. The checkpoint was trained around its attention design; serving software must support it efficiently.

Practical guidance

1. Inspect num_attention_heads, num_key_value_heads, head dimension, cache dtype, layer types, attention windows, and architecture-specific latent or positional state.
2. Compute per-layer and per-GPU cache bytes from the actual layout; include sharding, replication, allocator blocks, and non-KV memory.
3. Measure KV capacity separately from K/V traffic and attention-kernel time.
4. Benchmark representative context lengths, concurrency, input/output distributions, runtime versions, kernels, parallel topology, TTFT, ITL/TPOT, throughput, and goodput.
5. Re-run task and safety evaluations. Published MQA, GQA, or MLA results do not substitute for the checkpoint and workload you will serve.
6. If KV capacity is not binding, do not assume a different attention architecture will improve capacity-driven concurrency. The project's L4 sweep did not identify the limiting hardware resource.

Summary

• For conventional MHA, GQA, and MQA with equal head dimensions and precision, the attention KV-cache component scales linearly with K/V-head count.
• Past query vectors are not cached, but query-head computation is not free.
• Fewer K/V heads guarantee less conventional attention-cache state—not higher end-to-end throughput or a universal quality outcome.
• GQA is common in prominent open families; it is not the only modern attention design.
• MLA caches a learned compressed K/V latent plus positional key state and may use different prefill- and decode-oriented execution paths.
• Capacity occupancy cannot diagnose bandwidth pressure or the hardware bottleneck.

Knowledge check

A model config says “32 query heads, 8 key/value heads.” What can you infer about its attention cache?

This is conventional GQA: four query heads share each K/V head. Holding K/V head dimension, cache precision, layers, batch, and sequence length equal, its attention KV-cache component is 8/32 = one quarter of the corresponding 32-K/V-head MHA design. That ratio does not include non-KV memory, sliding windows, architecture-specific state, or runtime allocation overhead, and it does not by itself predict end-to-end speed or quality.

Holding model size, training quality, context, runtime support, and other architecture choices comparable, what would you expect from MQA versus MHA—and what must still be benchmarked?

MQA uses one shared K/V head, so it should require less conventional attention KV memory and read fewer K/V bytes than MHA. Whether that yields more admitted sequences, lower ITL, higher throughput, or lower serving cost depends on which resource binds and whether the runtime has efficient kernels. Benchmark per-GPU memory, queueing, TTFT, ITL/TPOT, throughput, goodput, and quality on the intended workload before claiming the economic result.

Why does 1.4% KV-cache occupancy rule out one limit but not establish that K/V traffic is unimportant?

It rules out exhaustion of the measured KV capacity at that load. Occupancy is not a bandwidth counter: the stored state may still be read on every decode step and contribute meaningful traffic or kernel time. Compare hardware counters, kernel profiles, and controlled architecture or context changes before attributing performance.

Primary sources

• Attention Is All You Need — the original multi-head attention architecture.
• Fast Transformer Decoding: One Write-Head Is All You Need — MQA and its controlled decoding/quality results.
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — the GQA definition and controlled uptraining evidence.
• FlashAttention-2 — efficient attention kernels and MQA/GQA support.
• DeepSeek-V2 — MLA, compressed K/V state, and decoupled positional treatment.
• vLLM MLA attention implementation — compute-friendly and data-movement-friendly paths.
• Introducing Meta Llama 3 and Mistral 7B — concrete GQA adoption examples.

Related chapters

• The KV cache and context growth — cache accounting and physical allocation
• Transformer and attention intuition — where query, key, and value projections come from
• Prefill and decode — phase behavior and regime-dependent bottlenecks
• Batching and concurrency — when memory, scheduler, and workload limits bind
• Quantization — cache precision as a separate representation lever