Part 6 — Inference Fundamentals
Quantization
Understand the single most universal inference optimization: representing weights (and sometimes activations and the KV cache) in fewer bits. Learn why floating-point beats integer for quality, why GPTQ/AWQ and FP8 are different kinds of thing, what is safe to quantize, and why you must always re-measure quality afterward.
Quantization is the one optimization that helps almost everything: it moves fewer bytes, so it speeds decode, frees VRAM for bigger batches, and — in the right format — accelerates prefill too. It is also the one most likely to quietly cost you quality. This chapter is about getting the wins without paying the hidden bill.
What you will understand by the end
- Why lower precision helps all three resources — compute, bandwidth, and capacity.
- Why floating-point beats integer at the same bit-width (dynamic range).
- Why GPTQ/AWQ and FP8 are not the same category of thing.
- What is safe to quantize, and why you always measure quality afterward.
Core concept: fewer bits per value
Recall from Prefill and decode that a model's footprint is
params × bytes-per-param. Quantization lowers the bytes per value (FP16 → FP8 →
INT4), and that pays off three ways at once:
- Compute: on a card with native low-precision tensor cores, prefill runs at roughly 2× the FLOPS per precision step.
- Bandwidth: decode streams half the bytes, so it runs proportionally faster.
- Capacity: the smaller model frees VRAM for a bigger KV-cache budget — more concurrent requests.
It is not a clean 2×. Scale factors and conversions add overhead, so each precision step is more like 30–50% faster (FP16→FP8 ≈ 1.3–1.5×). Quote it honestly — "roughly a third to a half faster per step," not "double."
Precision errors compound. Round π to 3.14 and π³ is off in the third digit; round it to the integer 3 and π³ collapses from 31 to 27. Inference chains thousands of operations, so small rounding errors accumulate. Most of quantization engineering is containing that.
Why floating-point beats integer
Every number format trades three things: precision (bits), dynamic range (how far apart the largest and smallest representable values are), and granularity (how many values share one scale factor).
A float is sign + exponent + mantissa; the exponent buys high dynamic range, so it faithfully represents the rare large values ("outliers") that carry disproportionate signal. An integer has no exponent → low dynamic range → it crushes outliers. So quality-sensitive production inference uses a float format (FP8, e.g. E4M3), not integer.
Finer granularity — a scale factor per block of values rather than per whole tensor — preserves outliers better at the cost of storing more scale factors. Recent "microscaling" formats (MXFP8, NVFP4) apply a scale every 16–32 values and hide the overhead in the tensor cores, which is how 4-bit floating-point becomes viable.
Algorithm vs format: GPTQ, AWQ, FP8
These are constantly listed side by side as if they were alternatives. They are not the same kind of noun:
| GPTQ | AWQ | FP8 | |
|---|---|---|---|
| What it is | an algorithm | an algorithm | a number format |
| Target | INT4 | INT4 | FP8 (float) |
| Quantizes | weights only | weights only | weights + activations + KV |
| Memory win | yes | yes | yes |
| Compute win | no | no | yes (native) |
GPTQ and AWQ are clever algorithms for surviving INT4 — GPTQ uses second-order error compensation, AWQ protects the ~1% of weights the activations care most about. But they are weight-only: the INT4 weights are dequantized back to FP16 to do the matmul, so you get the memory win (fit a big model on a small GPU) but no compute win. FP8 is a format the tensor cores read natively, so it also accelerates prefill.
The decision follows directly:
- GPTQ / AWQ (INT4) — when you must fit a big model on small/cheap VRAM, and the workload is memory-bound and latency-oriented (compute speed isn't the constraint).
- FP8 — for datacenter production, especially compute-bound / prefill-heavy work where you want the tensor-core acceleration, on hardware with native FP8. (On the newest cards the 4-bit answer is a float — NVFP4/MXFP4 — not INT4.)
What is safe to quantize: the sensitivity ladder
Not all of a model tolerates it equally. From least to most sensitive:
| Component | Practice |
|---|---|
| Weights (linear layers) | quantize freely (but leave the input/output layers full) |
| Activations | usually fine |
| KV cache | often worth it (stores more, reads faster — helps caching), but errors compound token-to-token |
| Attention / softmax | almost never — sensitive and compounding |
Quantize weights → activations → maybe the KV cache; leave attention and softmax at full precision. Larger models tolerate quantization better than small ones.
Quantize, then measure — always
Quantization is a dial, not a switch (FP8 not FP4, weights-only not everything), and its cost is invisible until you look. Measure quality after every quantization decision, cheapest to most trustworthy: perplexity (model "surprise" on known text) → standard benchmarks → a custom eval on your own task (the gold standard).
"Quantize, then measure" is not a slogan — it is exactly the discipline the evaluation half of this book teaches. The right target is a task-specific eval with a frozen dataset and exact scoring, because a quantized model can pass perplexity and still fail your task. See how to evaluate a configuration properly →
Common mistakes
- Reaching for INT4 by default. Integer crushes outliers; prefer a float format (FP8, or NVFP4 for 4-bit) unless you specifically need INT4's smaller size to fit a model on limited VRAM.
- Expecting a compute speedup from GPTQ/AWQ. They're weight-only — memory win only; the matmul still runs in FP16.
- Slamming the whole model to low precision. Respect the sensitivity ladder; leave attention/softmax full.
- Shipping without re-evaluating. The three wins are real; the quality cost is silent until measured.
Practical guidance
- Default to FP8 for datacenter serving on capable hardware; reach for INT4 (GPTQ/AWQ) only to fit a large model on small/cheap VRAM.
- Quantize weights first, then activations, then consider the KV cache; keep attention/softmax full precision.
- Treat quantization as weights → activations → KV dials you can turn independently, not one switch.
- After any change, run a task-specific eval and accept only if the drop is indistinguishable from run-to-run noise.
Summary
- Quantization moves fewer bytes → wins on compute, bandwidth, and capacity, but ~30–50% per step, not a clean 2×, and errors compound.
- Float beats integer at equal bits (dynamic range preserves outliers) → prefer FP8/NVFP4 for quality-sensitive work.
- GPTQ/AWQ are INT4 weight-only algorithms (memory win only); FP8 is a format with a native compute win.
- Follow the sensitivity ladder, and always re-measure quality on your own task.
Knowledge check
A teammate wants to quantize to INT4 with GPTQ to speed up a prefill-heavy workload. Two things are wrong — what?
(1) INT4 is an integer format with low dynamic range that crushes outliers — for quality-sensitive work a float format (FP8/NVFP4) is safer. (2) GPTQ is weight-only: the INT4 weights are dequantized to FP16 for the matmul, so it gives a memory win but no compute speedup — useless for a prefill/compute bottleneck. FP8 (native) is the right call there.
Why must you always measure quality after quantizing, and what's the best measurement?
Because precision errors compound and the loss is invisible until tested — a model can pass perplexity and still fail your task. The best measurement is a custom, task-specific eval on a frozen dataset with exact scoring, not a generic benchmark.