Part 6 — Inference Fundamentals
Quantization
Understand the single most universal inference optimization: representing weights (and sometimes activations and the KV cache) in fewer bits. Learn what quantization reliably buys you (memory) versus what it only sometimes buys you (speed), why floating-point and integer formats trade differently, why GPTQ/AWQ and FP8 are different kinds of thing, and why you must always re-measure both speed and quality on your own configuration.
Quantization is the one optimization that touches almost everything: fewer bytes per value means a smaller footprint, less memory traffic, and room for more concurrent requests. What it does to speed is much more conditional than its reputation suggests — and it is the optimization most likely to quietly cost you quality. This chapter is about knowing which wins are reliable, which ones you have to measure, and what the hidden bill looks like.
What you will understand by the end
- What quantization reliably buys (capacity, memory traffic) versus what is workload-dependent (latency and throughput).
- Why float and integer formats trade differently at the same bit-width.
- Why GPTQ/AWQ and FP8 are not the same category of thing — and why "weight-only" says something about arithmetic, not about performance.
- What is safe to quantize, and why you always measure quality afterward.
Core concept: fewer bits per value
Recall from Prefill and decode that a model's footprint is
params × bytes-per-param. Quantization lowers the bytes per value (FP16 → FP8 →
INT4). Two of the consequences are close to guaranteed; one is not:
- Capacity (reliable). The smaller model frees VRAM for a bigger KV-cache budget — more concurrent requests, or a model that simply wouldn't have fit.
- Memory traffic (reliable). Fewer bytes have to move from HBM to the compute units for each forward pass.
- Speed (conditional). Less traffic and lower-precision arithmetic can translate into lower latency or higher throughput. Whether they do — and by how much — depends on the hardware, the kernel, the runtime, the batch shape, and which resource you were actually bound by.
Whether it's faster depends on what you're bound by
This is the distinction that makes the rest of the chapter make sense. A forward pass is limited either by memory bandwidth (waiting on weights and KV to arrive) or by compute (the tensor cores are saturated):
- Bandwidth-bound work — chiefly small-batch decode, where you stream the whole weight set to generate one token per sequence. Here reducing bytes attacks the actual bottleneck, and quantization tends to help.
- Compute-bound work — chiefly large-batch prefill, where the arithmetic dominates. Here reducing stored bytes doesn't address the bottleneck; you only gain if the format also gives you faster arithmetic, and you can even lose ground to conversion overhead.
Published quantization speedups are measured on a specific model, card, kernel, runtime, and batch shape — and they do not transfer. Halving the bits does not halve the time: scale factors, dequantization, and conversions all cost something, and if you weren't bandwidth-bound to begin with, the traffic you saved wasn't what was slowing you down. Treat any single number — "2×", "30–50%", any figure quoted in this chapter or elsewhere — as a claim about someone else's configuration. The only number that describes yours is the one you benchmark on your target workload.
Precision errors compound. Round π to 3.14 and π³ is off in the third digit; round it to the integer 3 and π³ collapses from 31 to 27. Inference chains thousands of operations, so small rounding errors accumulate. Most of quantization engineering is containing that.
How float and integer formats differ
Every number format trades three things: precision (bits), dynamic range (how far apart the largest and smallest representable values are), and granularity (how many values share one scale factor).
A float is sign + exponent + mantissa; the exponent buys high dynamic range, so it represents the rare large values ("outliers") that carry disproportionate signal without crushing the small ones. An integer has no exponent, so a single scale must cover the whole range of values it applies to — and a few large outliers can consume that range at the expense of everything else. At equal bit-width, this makes float formats the more forgiving starting point.
Forgiving is not the same as required. Granularity is the other half of the story: a scale factor per small block of values, rather than per tensor, means an outlier only distorts its own block. This is why well-implemented INT4 with fine-grained grouping remains viable — and widely deployed — for quality-sensitive work, and why the naive framing "integer is for when you don't care about quality" is wrong. Recent "microscaling" float formats (MXFP8, NVFP4) apply a scale every 16–32 values and hide the overhead in the tensor cores, which is how 4-bit floating-point became practical too.
The honest summary: float buys you more headroom per bit, fine granularity buys back much of what integer gives up, and which combination is best for your model and hardware is an empirical question — not one the format names settle.
Algorithm vs format: GPTQ, AWQ, FP8
These are constantly listed side by side as if they were alternatives. They are not the same kind of noun:
| GPTQ | AWQ | FP8 | |
|---|---|---|---|
| What it is | an algorithm | an algorithm | a number format |
| Target | INT4 | INT4 | FP8 (float) |
| Quantizes | weights only | weights only | weights + activations + KV |
| Memory win | yes | yes | yes |
| Lower-precision arithmetic | no (W4A16) | no (W4A16) | yes (native tensor cores) |
| Speedup regime | mainly bandwidth-bound work | mainly bandwidth-bound work | bandwidth- and compute-bound |
GPTQ and AWQ are clever algorithms for surviving INT4 — GPTQ uses second-order error compensation, AWQ protects the ~1% of weights the activations care most about. Both are weight-only (W4A16): activations stay at higher precision, so the multiply-accumulate itself is not done in 4-bit. FP8 is a format the tensor cores read natively, so it can accelerate the arithmetic as well.
"Weight-only means no lower-precision arithmetic" is true. "Weight-only means no performance win" does not follow, and it is the more common error. An optimized weight-only kernel reads compressed weights from memory and expands them near the compute units — so it moves far fewer bytes. When the matmul is memory-bandwidth-bound — which small-batch decode usually is — that is an attack on the actual bottleneck, and W4A16 GPTQ/AWQ can improve latency and throughput meaningfully. When the matmul is compute-bound, the same kernel has nothing to gain and the dequantization overhead can make it slower than FP16. The benefit is real, and it is conditional on the regime, the kernel, and the hardware.
The decision follows — as a starting hypothesis to benchmark, not a rule:
- GPTQ / AWQ (INT4) — strongest when you must fit a big model on small or cheap VRAM, and when the workload is bandwidth-bound (small-batch, latency-oriented decode), where reading compressed weights is a direct win. Weaker, and sometimes negative, on compute-bound prefill.
- FP8 — attractive for datacenter production on hardware with native FP8 support, because it can help in both regimes rather than only the bandwidth-bound one. (On the newest cards the 4-bit answer is often a float — NVFP4/MXFP4 — rather than INT4.)
Neither line is a default. Both describe where a configuration is likely to pay off, on the way to measuring whether it actually does.
What is safe to quantize: the sensitivity ladder
Not all of a model tolerates it equally. From least to most sensitive:
| Component | Practice |
|---|---|
| Weights (linear layers) | quantize freely (but leave the input/output layers full) |
| Activations | usually fine |
| KV cache | often worth it (stores more, reads faster — helps caching), but errors compound token-to-token |
| Attention / softmax | almost never — sensitive and compounding |
Quantize weights → activations → maybe the KV cache; leave attention and softmax at full precision. Larger models tolerate quantization better than small ones.
Quantize, then measure — both halves
Quantization is a dial, not a switch (FP8 not FP4, weights-only not everything), and it has two costs that are invisible until you look. Measure both, on the configuration you actually intend to ship:
- Quality, cheapest to most trustworthy: perplexity (model "surprise" on known text) → standard benchmarks → a custom eval on your own task (the gold standard).
- Speed, on your target hardware, runtime, kernel, and batch shape — because as the sections above showed, the same quantization can be a large win, a wash, or a regression depending on which resource you were bound by. Benchmark against the unquantized baseline on the same rig; a speedup someone else measured is a hypothesis, not a result.
"Quantize, then measure" is not a slogan — it is exactly the discipline the evaluation half of this book teaches. The right target is a task-specific eval with a frozen dataset and exact scoring, because a quantized model can pass perplexity and still fail your task. See how to evaluate a configuration properly →
Common mistakes
- Assuming a speedup transfers. A number measured on another model, card, kernel, or batch shape tells you almost nothing about yours.
- Reading "weight-only" as "no performance benefit." W4A16 does no 4-bit arithmetic, but reading compressed weights is a genuine win on bandwidth-bound decode.
- Expecting weight-only INT4 to accelerate compute-bound prefill. That's the regime where it has nothing to attack and dequantization overhead can make it slower.
- Treating format names as quality verdicts. Integer with fine-grained grouping is deployed in quality-sensitive production; float buys headroom, not immunity.
- Slamming the whole model to low precision. Respect the sensitivity ladder; leave attention/softmax full.
- Shipping without re-evaluating. The memory win is nearly automatic; the speed win isn't, and the quality cost is silent until measured.
Practical guidance
- Choose the starting configuration from what you're bound by and what your hardware supports natively — not from a default. Bandwidth-bound decode on constrained VRAM points toward weight-only INT4; native FP8 support and mixed prefill/decode traffic point toward FP8.
- Benchmark the exact target configuration — model, card, kernel, runtime, batch shape — against an unquantized baseline before believing any speedup.
- Quantize weights first, then activations, then consider the KV cache; keep attention/softmax full precision.
- Treat quantization as weights → activations → KV dials you can turn independently, not one switch.
- After any change, run a task-specific eval and accept only if the drop is indistinguishable from run-to-run noise.
- Re-benchmark after runtime and kernel upgrades. Weight-only kernels in particular have improved substantially over time, so an old measurement can under-sell a current option.
Summary
- Quantization reliably reduces capacity requirements and memory traffic. Whether it improves latency or throughput depends on hardware, kernels, runtime, batch shape, and whether the workload was bandwidth- or compute-bound.
- Float and integer trade differently at equal bits — float has more dynamic range, fine-grained grouping recovers much of the difference for integer — and neither name settles the quality question.
- GPTQ/AWQ are INT4 weight-only algorithms: no 4-bit arithmetic, but a real speedup where memory bandwidth is the bottleneck. FP8 is a format the tensor cores read natively, so it can help in both regimes.
- Follow the sensitivity ladder, benchmark the target configuration, and always re-measure quality on your own task.
Knowledge check
A teammate wants to quantize to INT4 with GPTQ to speed up a prefill-heavy workload. What's the problem — and what would change your answer?
The problem is the regime, not the algorithm. Prefill at reasonable batch sizes is typically compute-bound, and GPTQ is weight-only (W4A16): the multiply-accumulate still happens at higher precision. Reading compressed weights doesn't attack a compute bottleneck, and the dequantization overhead can leave you slower than FP16. On hardware with native FP8, a format the tensor cores execute directly is the more promising candidate here.
Note what is not wrong with the plan. "INT4 is integer, so quality will suffer" is too strong — with fine-grained grouping and a good algorithm, INT4 is deployed in quality-sensitive production. And if the workload were small-batch decode instead, the same GPTQ setup would be a sensible first thing to try, because bandwidth is the bottleneck there and compressed weights address it directly.
What would change the answer: measured evidence. Batch shape, kernel implementation, and card all move this, so the honest last step is to benchmark both candidates on the target configuration rather than reasoning it out.
Why must you always measure quality after quantizing, and what's the best measurement?
Because precision errors compound and the loss is invisible until tested — a model can pass perplexity and still fail your task. The best measurement is a custom, task-specific eval on a frozen dataset with exact scoring, not a generic benchmark.