Inference runtimes

The inference engine is the piece of the stack you actually operate. The frameworks below it (Model formats and frameworks) package the weights; the engine serves them — batching requests, managing the KV cache, and wiring in the hand-written kernels (CUDA and kernels) that make LLM inference fast. Crucially, every technique you've learned shows up here as a flag, so this is where theory becomes a running system.

What you will understand by the end

• The one axis that separates the engines: breadth vs peak performance.
• Why choosing an engine is not choosing which techniques you get.
• How each optimization maps to an engine flag — the map the whole stack was building to.
• A decision procedure for picking an engine for a real workload.

The organizing principle

The more constraints you accept, the better performance you can reach.

That sentence is the entire decision axis. Broadening what an engine supports — more hardware, more models, day-zero — costs peak performance; narrowing the target buys it back.

   BROAD / convenient ◄──────────────────────────────────────► NARROW / fast
   (fewer constraints)                                         (more constraints)

   vLLM ─────────────────── SGLang ─────────────────── TensorRT-LLM
   most hardware & models    NVIDIA+AMD; MoE &          NVIDIA-only, Hopper+;
   day-zero, easiest         multi-node specialist       some models; hardest,
   "good" performance        "good" performance          "best" performance

Engine	Performance	Ease	Model support	Hardware
vLLM	Good	Easy	Most	NVIDIA / AMD / Intel + TPU
SGLang	Good	Easy	Most	NVIDIA + AMD (MoE & multi-node specialist)
TensorRT-LLM	Best	Hard	Some	NVIDIA only (Hopper+)

The map everything was building to: technique → flag

This is the payoff of the whole stack. Each optimization is a switch on the engine — you already understand the left column; the engine just teaches you where the switch is (flag names are representative; each engine has equivalents):

Technique	What it does	Typical flag (vLLM)
Continuous batching	reuse one weight-load across concurrent requests	on by default
Quantization	move fewer bytes per pass	--quantization fp8, --kv-cache-dtype fp8
PagedAttention	pack the KV cache to fit more requests	on by default
Prefix caching	reuse KV for shared prompt prefixes	--enable-prefix-caching
Chunked prefill	interleave prefill chunks with decode	--enable-chunked-prefill
Speculative decoding	spend idle compute to save memory trips	--speculative-config …
Parallelism	shard a too-big model across GPUs	--tensor-parallel-size N
Batch ceiling	cap the running batch	--max-num-seqs N

Under the hood: when the CPU is the bottleneck

Not all of an engine's speed is GPU kernels. As GPUs get faster, the host (CPU) side of each decode step — the scheduler deciding the batch, launching kernels, and sampling the next token — can become the limit, especially at small batch, where the GPU finishes so quickly it ends up waiting on the CPU. Two runtime tricks fight this, and they are a big part of why a well-built engine stays fast at low batch:

• CUDA graphs. Rather than launching each kernel individually every step (hundreds of small CPU→GPU launches), the engine captures the decode step's kernel sequence once and replays it as a single unit, erasing most launch overhead. (In vLLM this is on by default; --enforce-eager disables it for debugging.)
• Multi-step scheduling. Run several decode steps per scheduler invocation, so the CPU-side scheduling and sampling cost is amortised across many tokens instead of paid on every single one.

The engines, briefly

• vLLM — the broad default. Oldest and most widely used; runs on the most hardware (including AMD, Intel, and TPU) and supports the most models day-zero. Impressive performance when well-configured, but below TensorRT-LLM's ceiling — breadth costs peak. The natural place to stand up almost any model quickly. Multimodal inputs via its Omni variant.
• SGLang — the MoE / multi-node specialist. A fast backend plus a flexible frontend for deep customization; strongest on large mixture-of-experts models and multi-node serving. The choice when you're serving a big MoE cost-efficiently.
• TensorRT-LLM — the performance ceiling. NVIDIA's engine, fastest via its own (partly closed-source) kernels and native low-precision formats on Hopper/Blackwell. Best throughput-per-dollar when the architecture is well-supported — at the cost of a harder setup and NVIDIA-only lock-in. (Watch the version: its V1 is standalone and PyTorch-based; older V0 is a TensorRT plugin.)

A decision procedure

1. What are you serving? Image/video generation → a diffusion path, not an LLM engine. Multimodal understanding → vLLM Omni or SGLang multimodal. Text LLM → continue.
2. Hardware and model support. Non-NVIDIA hardware or a brand-new model needing day-zero → vLLM. Large MoE, throughput-first, multi-node → SGLang. Well-supported model on Hopper+ and willing to invest engineering for peak → TensorRT-LLM.
3. Prototype, then measure. Stand up on vLLM to prove the workload runs anywhere, then benchmark a peak-performance engine only if the delta justifies the engineering — decided by a head-to-head benchmark, never the marketing line.

Common mistakes

• Thinking the engine choice limits your techniques. It doesn't — they all share the menu. It limits performance ceiling, hardware, and model support.
• Reaching for the "fastest" engine before confirming model support. "Best performance" is worthless if the architecture doesn't run there.
• Tuning many flags at once. Change one, measure, keep or revert — the discipline the experiments live by.
• Over-aggressive batch/memory flags. Push --max-num-seqs or memory utilization too far and requests get preempted or the server OOMs; watch KV-cache utilization.

Practical guidance

• Treat a deployment as engine + model + a flag-set, and manage that flag-set as config-as-code — it's what you tune in a benchmark loop.
• Scrape the engine's metrics endpoint (TTFT, tokens/sec, running-vs-waiting requests, KV-cache utilization, prefix-cache hit rate) — that's your SLO dashboard and your first diagnostic when latency drifts.
• Use official container images so CUDA, drivers, and tuned kernels arrive matched.
• Default to vLLM for the prototype; earn your way to TensorRT-LLM with a measured win.

Summary

• Engines sit on one axis: breadth (vLLM) ↔ peak performance (TensorRT-LLM), with SGLang the MoE/multi-node specialist.
• They share the technique menu, so you choose performance/breadth/hardware/model-fit — not which optimizations you get.
• Every technique is a flag, and this project's experiments are that map measured on a real endpoint.
• Prototype broad, then benchmark your way to peak.

Knowledge check

All three engines support quantization, prefix caching, parallelism, and speculation. So what are you actually choosing between when you pick one?

Not which techniques are available — those are common. You're choosing peak performance vs breadth/ease, which hardware you can run on, and whether this specific model is well-supported (ideally day-zero) on that engine — plus operational fit (multi-node MoE → SGLang, run-anywhere/new model → vLLM, peak on Hopper+ → TensorRT-LLM).

You need to ship a brand-new open model this week on whatever GPUs you can get. Which engine, and why?

vLLM: it has the broadest hardware support and is usually first to day-zero support for new models, with the easiest stand-up. You can prove the workload immediately, then evaluate a peak-performance engine later if a benchmark justifies the extra engineering.

Related evidence

• The technique→flag map, measured one knob at a time →
• Continuous batching, the default that makes an engine fast →
• Forward: orchestrating engines across many nodes →