vLLM vs SGLang vs TensorRT-LLM

An inference engine is the runtime you operate around model execution. It schedules requests, forms iteration batches, manages KV state, dispatches kernels, applies precision and parallelism choices, and exposes serving controls and metrics. This is where many ideas from batching, KV caching, and CUDA kernels become concrete configuration.

Choosing among vLLM, SGLang, and TensorRT-LLM is not choosing a permanent first-, second-, and third-place engine. It is a compatibility filter, followed by an operational fit check, followed by a workload-specific benchmark.

What you will understand by the end

• What an inference engine controls and what remains hardware-, model-, and version-specific.
• How major optimization categories map to representative configuration.
• Why shared feature labels do not imply equal maturity or interchangeable implementations.
• The current, dated reasons each major engine enters a benchmark shortlist.
• How to compare engines using the same workload, SLO, topology, and measurement method.

The organizing principle: filter, then benchmark

Breadth versus specialization is one useful first lens, not the entire decision axis. Selection is multi-dimensional:

1. Compatibility: exact model, modality, hardware, precision, feature combinations, and distributed topology.
2. Operational fit: deployment, observability, failure recovery, ecosystem integration, portability, release cadence, and team expertise.
3. Measured workload fit: prompt/output lengths, concurrency, prefix reuse, latency SLO, quality, memory, throughput, stability, and cost.

What the current shortlist looks like

Engine capabilities change quickly. The table is an orientation reviewed on 2026-08-02, not a universal ranking:

Engine	Why it enters the shortlist	Hardware orientation	What must be verified
vLLM	Broad ecosystem, model compatibility, hardware reach, and a low-friction initial serving path	Broad native and plugin-based coverage; platform support varies	Exact model and modality maturity, platform-specific features, distributed behavior, benchmark result
SGLang	Strong prefix-reuse, structured-program, MoE, expert-parallel, and distributed-serving capabilities	Documented support includes NVIDIA, AMD, Intel Xeon, Google TPU, and Ascend	Hardware-specific maturity, topology, feature combinations, operations, benchmark result
TensorRT-LLM	Aggressive NVIDIA optimization, low-precision and distributed features, and current PyTorch execution path	NVIDIA-only; current matrix includes Ampere, Ada, Hopper, and Blackwell families, with architecture-dependent support	Exact model/GPU/precision combination, beta versus stable paths, release compatibility, measured benefit

No “Good” or “Best” label survives without a workload definition. TensorRT-LLM may be the strongest result for one supported NVIDIA configuration; vLLM or SGLang may lead for another model, topology, platform, or release.

Current TensorRT-LLM boundary

Current TensorRT-LLM uses PyTorch as its sole execution backend and loads supported Hugging Face checkpoints directly. The legacy TensorRT-engine build and checkpoint- conversion path has been removed from the current release. Older tutorials using trtllm-build, convert_checkpoint.py, or backend="tensorrt" describe a legacy workflow.

Multimodal understanding and visual generation are separate workloads. TensorRT-LLM supports vision-language inference and now documents beta VisualGen support for diffusion-based image and video generation. Check the supported-model list and maturity of the exact path instead of treating visual generation as either universally present or absent.

Technique to configuration: a conceptual map

The same optimization categories appear across engines, but names, defaults, semantics, valid ranges, combinations, and maturity differ. The examples below use representative vLLM controls and are not portable configuration:

Technique	What it does	Representative vLLM control
Continuous batching	Dynamically admit ready requests and remove completed requests between iterations	enabled by the scheduler
Quantization	Reduce weight or cache representation bytes, subject to compatible kernels	--quantization …, --kv-cache-dtype …
Paged KV management	Allocate KV state in blocks to reduce fragmentation and reclaim incrementally; sharing/reuse where supported	enabled by the runtime
Prefix caching	Reuse retained KV blocks for matching tokenized prefixes	--enable-prefix-caching
Chunked prefill	Split long prefills and co-schedule chunks under a token budget	--enable-chunked-prefill
Speculative decoding	Propose draft tokens cheaply and verify them with the target, reducing serial target steps when acceptance is high enough	--speculative-config …
Parallelism	Distribute model, requests, experts, or sequence work to meet fit, latency, throughput, or scale goals	--tensor-parallel-size N plus topology-specific controls
Sequence ceiling	Bound sequences scheduled per iteration	--max-num-seqs N
Token ceiling	Bound tokens scheduled per iteration	--max-num-batched-tokens N

Confirm the current engine's exact option, default, valid feature combinations, and emitted metrics before translating a setting. A category checkbox is especially weak evidence: vLLM currently labels disaggregated prefill experimental and states that it does not improve throughput, while other engines have their own maturity and topology boundaries.

What the one-knob experiment established

Runtime overhead beyond GPU kernels

Engine performance also includes host scheduling, sampling, graph construction or replay, memory allocation, communication, and request bookkeeping. At small batches or with very fast kernels, host launch and scheduling overhead can become material; it is not guaranteed to dominate.

• CUDA graphs can capture a compatible kernel-launch sequence and replay it with less host overhead. Dynamic shapes, unsupported operations, memory requirements, and engine policy constrain when capture applies.
• Multi-step or asynchronous scheduling can amortize or overlap CPU-side work across decode steps, with engine-specific behavior and latency trade-offs.

These are additional reasons two engines can differ on the same GPU. Profile before naming the host as the bottleneck.

Build the shortlist in three steps

1. Compatibility filter

• Does this version support the exact architecture and checkpoint path?
• Does it support the required text, VLM, embedding, audio, or visual-generation workload?
• Does it support the available hardware, interconnect, driver, and architecture generation?
• Are the required precision, quantization, cache, speculation, and structured-output paths valid together?
• Is the required single- or multi-node topology supported and mature?

2. Operational filter

• Deployment and upgrade complexity
• Metrics, traces, debugging, and failure recovery
• API and ecosystem integration
• Portability and vendor constraints
• Team familiarity and maintenance capacity
• Release cadence and support expectations

3. Controlled benchmark

Run surviving engines with the same:

• model weights and quality checks;
• precision and quantization scheme;
• hardware, topology, and replica count;
• input/output-length, concurrency, burst, and prefix-reuse distributions;
• feature configuration and cache warm-up;
• TTFT, ITL/TPOT, end-to-end and tail-latency SLOs;
• warm-up, sample count, duration, repetitions, and measurement boundary.

Report TTFT, ITL/TPOT, output tokens/s, request throughput, p50/p95/p99 latency, goodput, GPU and KV memory, queueing, cache behavior, preemption/OOM/errors, and cost per SLO-compliant request or per output-token volume. A peak throughput number without its latency constraint cannot select a serving engine.

A realistic decision example

Suppose you need to serve a Qwen MoE checkpoint on eight H200s with strict p99 ITL, long shared system prompts, and possible multi-node expansion. Do not select “SGLang for MoE” or “TensorRT-LLM for speed” from a generic table. First confirm current model, quantization, expert-parallel, prefix-cache, and topology support. Then benchmark SGLang, TensorRT-LLM, and vLLM against the same request trace and SLO. The best result is the one that meets quality and operational requirements with the strongest measured goodput or cost—not the engine with the strongest general reputation.

Common mistakes

• Treating the table as a speed ranking. It is a dated shortlist orientation.
• Assuming shared feature labels mean parity. Stability, semantics, supported combinations, and performance vary by model, platform, and version.
• Using a deterministic assignment such as “large MoE → SGLang.” Put plausible engines on the shortlist, then test the exact topology.
• Assuming day-zero support. vLLM is often a low-friction first compatibility check for a new Hugging Face model, but every engine's model matrix changes.
• Calling continuous batching weight reuse. Its defining behavior is dynamic admission and removal at scheduling boundaries.
• Tuning many controls at once. Change one dimension, measure, and retain or revert.
• Ignoring host-driver compatibility. A GPU container packages user-space libraries and engine dependencies; the host supplies the NVIDIA driver.

Practical guidance

1. Manage the complete deployment as versioned configuration: engine, image, model revision, runtime libraries, precision, kernels, topology, flags, and environment.
2. Prefer tested official images to align engine dependencies, user-space CUDA libraries, and packaged kernels. Validate the host GPU driver separately against the image's minimum requirements.
3. Scrape TTFT, ITL/TPOT, request/output throughput, running/waiting work, KV use, cache hits, preemption, errors, and OOMs. Keep engine-specific metric definitions with the dashboard.
4. Start with the option that clears compatibility and minimizes operational friction for your team—not a universal default. Add alternatives when the expected benchmark value justifies the comparison cost.
5. Re-run the comparison after material engine, model, kernel, driver, or workload changes.

Summary

• vLLM, SGLang, and TensorRT-LLM cannot be placed in a permanent speed order.
• Filter by exact compatibility and feature maturity, assess operational fit, then benchmark the actual workload and SLO.
• Major feature categories overlap, but implementations and valid combinations are not interchangeable.
• Current TensorRT-LLM uses a PyTorch execution backend; its legacy TensorRT-engine build path is removed. Its NVIDIA support spans more than Hopper and Blackwell.
• Continuous batching, paged KV management, speculation, and parallelism have narrower definitions than a checkbox or flag name suggests.
• A reproducible engine result includes software version, model, precision, hardware, topology, request distribution, features, quality, latency, throughput, reliability, and cost.

Knowledge check

Why is “TensorRT-LLM is fastest” not enough to select it?

“Fastest” is undefined without the engine version, model and weights, precision, NVIDIA GPU generation, topology, request distribution, enabled features, quality checks, and latency SLO. First confirm exact support and operational fit, then compare surviving engines under the same controlled workload. Another engine can lead on a different configuration.

Why does shared support for “disaggregation” or “speculative decoding” not make engine implementations interchangeable?

The label does not reveal stability, model and hardware coverage, valid feature combinations, topology, semantics, observability, setup cost, or measured benefit. For example, a path may be experimental, may target latency rather than throughput, or may work only with particular models and parallel layouts. Verify current documentation and benchmark the enabled path.

What is wrong with defining continuous batching as reusing one weight load across concurrent requests?

Weight-traffic amortization can be a benefit of batching, but it is not what makes batching continuous. Continuous or in-flight batching dynamically admits ready requests and removes completed requests between scheduling iterations instead of holding one fixed group until every request finishes.

Primary sources and version boundary

Product capabilities were reviewed against current documentation on 2026-08-02. Engine features and support matrices change rapidly; consult the current release documentation.

• vLLM supported models and parallelism and scaling — compatibility and distributed execution.
• vLLM disaggregated prefill — experimental status and stated performance objective.
• SGLang documentation and PD disaggregation — hardware and distributed-serving coverage.
• TensorRT-LLM support matrix and supported models — NVIDIA architecture, precision, and model coverage.
• TensorRT-LLM backend-removal guide — current PyTorch-only execution path.
• TensorRT-LLM VisualGen — beta image and video generation path.
• NVIDIA Container Toolkit installation — host-driver prerequisite.

Related chapters

• Model formats and frameworks — checkpoints and execution interfaces
• CUDA kernels for LLM inference — dispatch, compatibility, and kernel measurement
• Batching and concurrency — iteration batches, queues, and scheduling budgets
• NVIDIA Dynamo — distributed coordination around engine backends
• Latency, throughput and token rates — benchmark metrics and goodput