GPU architecture and inference hardware

A GPU spec sheet is not a ranking. It is a set of clues about where a particular inference workload may run out of headroom. The limiting resource can change between prefill and decode—and again with batch size, sequence length, precision, kernels, parallelism, and service-level objectives.

The useful question is therefore not “Which GPU is best?” It is: Which resource limits this workload, in this phase, on this serving stack?

What you will understand by the end

• The four hardware resources to investigate: compute, memory bandwidth, memory capacity, and interconnect.
• Why “prefill is compute-bound; decode is bandwidth-bound” is a common regime, not a law.
• How to compare actual SKU specifications without inferring capabilities from names.
• How to size memory from the workload instead of applying a fixed headroom percentage.
• What runtime telemetry can—and cannot—establish about a bottleneck.

Core concept: four resource hypotheses

Text generation has two phases. Prefill processes the prompt and produces the initial KV state. Decode generates subsequent tokens while reading weights and the growing KV cache. A useful first pass is:

PREFILL      → often compute-sensitive at long sequence lengths or efficient batch sizes
DECODE       → often bandwidth-sensitive for dense models at low batch sizes
CAPACITY     → weights + runtime memory + KV state for all resident tokens
INTERCONNECT → communication between devices when the model or workload is partitioned

These are starting hypotheses. In roofline terms, the result depends on arithmetic intensity: how much useful computation the system performs for each byte moved. A short or poorly batched prefill can remain memory-bound; a decode workload can move toward compute pressure as batching increases. Model architecture, attention length, quantization, kernel efficiency, and host overhead can move the boundary. Published measurements even disagree on whether large-batch decode crosses that boundary on a given setup—which is a reason to measure, not to pick a slogan. See the roofline analysis of LLM inference and a large-batch counterexample on H100.

A quick tour of the four resources:

• Compute throughput. Tensor cores execute the matrix operations. Lower-precision formats may offer greater peak throughput, but the ratio is specific to the GPU, data type, accumulation mode, sparsity assumption, and available kernel. “Half the bits means twice the speed” is not a hardware rule.
• Memory bandwidth. A low-batch dense decode often streams a large fraction of the weights for each token. In that narrow regime, bandwidth ÷ active weight bytes is a useful optimistic ceiling, not a throughput prediction. KV traffic, batching, mixture-of-experts routing, kernel overhead, and communication all lower or change it.
• Memory capacity. Device memory must hold weights, runtime workspaces, allocator overhead, and the KV state of resident requests. It constrains model fit, concurrent tokens, and therefore achievable batching.
• Interconnect. Tensor, pipeline, expert, and data parallelism move different data between devices. Link bandwidth and latency can dominate a distributed configuration even when each GPU has ample compute and memory.

CPU scheduling, kernel-launch overhead, storage, network ingress, or a latency SLO can also become the practical limiter. The four GPU resources are the hardware checklist, not a claim that nothing else matters.

Mental model: phases, regimes, and qualifications

Resource	Often matters when	What can change the conclusion
Compute	Long or efficiently batched prefill; sufficiently high arithmetic intensity	Sequence length, batch shape, precision, kernel utilization
Memory bandwidth	Low-batch dense decode; weight or KV reads dominate	Batch reuse, attention length, quantization, MoE sparsity, caching
Memory capacity	Large weights, long contexts, many resident requests	KV format, model architecture, allocator and runtime reservations
Interconnect	The model or request is partitioned across GPUs or nodes	Parallelism strategy, topology, collective size, overlap with compute

The bottleneck can move during one request. Prefill may stress compute while decode stresses memory traffic; increasing concurrency may improve compute utilization but consume the remaining KV capacity; splitting the model may solve capacity and introduce communication overhead. Hardware selection is therefore a workload-envelope decision, not a phase label.

Read the specifications, not the product name

There is no universal rule in which a product-name letter means compute generation and the number means memory size. Compare the actual fields for the exact SKUs under consideration.

For example, NVIDIA positions the H200 as a Hopper-family update with more and faster memory than H100. That is a useful specific comparison, not a naming grammar. Within Ada, the L4 and L40 differ materially in memory, bandwidth, and compute despite sharing a family letter.

When comparing peak throughput, normalize all of these:

• exact numeric format and accumulation mode;
• dense versus structured-sparsity figures;
• tensor-core versus non-tensor operations;
• clocks, power envelope, and form factor;
• whether the serving runtime has an efficient kernel for that path.

Peak arithmetic is an upper bound. It is not delivered token throughput.

Multi-GPU is a topology decision

NVLink can reduce communication cost between supported GPUs, but it does not turn several devices into one transparent GPU. The runtime still partitions weights or work, executes collectives or stage transfers, and manages per-device memory.

No NVLink is not automatically a dead end. It changes the viable parallelism and the cost. The vLLM parallelism guide specifically suggests pipeline parallelism instead of tensor parallelism when GPUs lack NVLink, using L40S as its example. Quantization, CPU offload, or another deployment shape may also be preferable. Benchmark the topology you will actually deploy.

Size memory from the workload

A fixed “weights plus 50%” allowance is too weak: two workloads using the same weights can have very different resident-token counts. Use a budget like:

required device memory
  = weight bytes
  + runtime and workspace bytes
  + KV bytes per token × peak resident tokens
  + measured safety margin

peak resident tokens includes the prompts and generated tokens of requests that coexist on the device. KV bytes per token depends on layers, KV heads, head dimension, numeric format, and whether the model uses grouped- or multi-query attention. Runtime reservations and fragmentation are measured properties, not universal percentages.

Then validate the estimate against the serving runtime. vLLM, for example, reports the GPU KV-cache token capacity and estimated maximum concurrency at startup. Those values are more actionable than “the model fits,” but they still need a load test at the intended sequence-length distribution.

What measurement can establish

Theory narrows the investigation. Counters and controlled experiments establish the cause.

High GPU or SM activity does not prove compute saturation. Active warps may be waiting on memory. NVIDIA's DCGM profiling guidance recommends reading SM activity alongside DRAM activity, tensor utilization, and other metrics. For a stronger diagnosis, combine counters with a controlled comparison: vary one relevant resource or workload dimension while holding the rest fixed.

The same project measured a large prefix-caching effect in its tested workload. That is evidence for the configuration, model, prompts, and runtime used—not a universal multiplier for every L4 deployment.

Common mistakes

• Turning the phase heuristic into a law. Prefill and decode can cross resource regimes as workload shape and implementation change.
• Comparing incompatible peak numbers. Precision, sparsity, accumulation, and kernel support must match before FLOPS figures are comparable.
• Inferring specifications from a SKU name. Product naming is not a capacity or compute formula.
• Using a fixed KV headroom percentage. Size from resident tokens and measured runtime overhead.
• Calling high GPU activity “compute-bound.” Activity establishes use, not the reason progress is limited.
• Treating missing NVLink as disqualifying. It may change the best parallelism strategy, but it does not make distributed inference impossible.

Practical guidance

1. Write down the workload envelope: model, precision, prompt and output distributions, concurrency, batch policy, latency targets, and serving runtime.
2. Form a bottleneck hypothesis for prefill and decode separately.
3. Calculate capacity from weights, runtime memory, and peak resident-token KV state.
4. If more than one GPU is required, evaluate the exact topology and parallelism strategy.
5. Benchmark representative traffic and inspect phase latency, queueing, KV capacity, DRAM activity, tensor utilization, and communication time.
6. Change one relevant dimension and repeat. A resource attribution is strongest when the predicted response appears under a controlled comparison.

Summary

• Compute, memory bandwidth, memory capacity, and interconnect are bottleneck hypotheses, not fixed phase assignments.
• Prefill is often compute-sensitive and low-batch dense decode often bandwidth-sensitive, but arithmetic intensity and implementation determine the actual regime.
• Compare exact specifications at compatible formats; product names and peak FLOPS are not performance predictions.
• Size memory from runtime overhead and resident-token KV demand, then verify the runtime's reported capacity under representative load.
• Low KV occupancy can rule out KV-capacity pressure. GPU activity alone cannot prove a compute bottleneck.

Knowledge check

A workload has long document inputs and one-line outputs. Can you choose between H100, H200, and B200 from that description alone?

No. Long inputs make compute pressure during prefill a reasonable hypothesis, but batch shape, sequence length, precision, kernels, latency targets, capacity requirements, and price all matter. Compare the exact SKU specifications at the same numeric format, then benchmark representative TTFT. H200's additional memory bandwidth and capacity help only if those resources constrain the chosen configuration.

Two GPUs list similar tensor throughput. What must match before the figures are meaningfully comparable?

At minimum: numeric format, accumulation mode, dense versus sparsity-assisted operation, and the kind of tensor operation being counted. Then check whether the runtime has an efficient kernel for each GPU and measure the real workload; equal peaks do not imply equal tokens per second or latency.

A server reports 1.4% KV-cache occupancy and high SM activity at its throughput knee. What can you conclude?

You can reasonably rule out KV-cache capacity as the limiter in that run. You cannot yet call it compute-bound: high SM activity can coexist with memory stalls or inefficient kernels. Inspect DRAM and tensor metrics and run a controlled comparison that should move the suspected limit.

Primary sources

• LLM Inference Unveiled: Survey and Roofline Model Insights — arithmetic intensity and phase-dependent regimes.
• Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference — a measured counterexample to a simple large-batch rule.
• NVIDIA DCGM profiling metrics — interpreting SM, tensor, and memory activity together.
• vLLM parallelism and scaling — topology-aware parallelism and runtime KV-capacity reporting.
• NVIDIA L4, L40, and H200 product pages — exact SKU specifications.

Related evidence

• The serving-knobs experiment — measured capacity and activity evidence on one L4 →
• The systems compared in this project and the hardware each used →