The Serving Knobs That Mattered - and the Ones That Didn't
Part 3: Turning one vLLM flag at a time on a live endpoint, to separate the settings that move throughput from the settings that only look important
In Part 1, I fixed a concurrency-propagation bug and raised throughput about 22x. In Part 2, I compared six models on the same intent contract and found that serving speed had never been the quality problem.
Both parts left the same loose end. The endpoint was fast, but I could not say which serving settings were doing the work. The original fast configuration had turned on prefix caching, chunked prefill, a batch-size cap, and a context-window limit all in the same deploy. Bundled changes prove that a configuration works. They do not tell you which knob to reach for next time.
So this part answers a narrower, more useful question:
If you change exactly one vLLM serving flag and hold everything else constant, what actually happens to throughput and tail latency?
I ran five isolated experiments against a live Qwen 2.5 3B endpoint on a single L4. The result was a clean ranking:
Prefix caching was worth 3-4x throughput on this workload. Lowering
max_num_seqsimposed a batch ceiling. Raisingmax_model_lenhad no measurable cost in the tested range. Chunked prefill did nothing measurable here.
Three of those are load-bearing. One is a genuine null result. Reporting the null one honestly is part of the point.
The Method: One Knob Per Deploy, Production Untouched
The trap in serving benchmarks is changing two things and claiming one effect. To avoid it, every experiment changed exactly one flag from the production baseline.
Each variant config was pushed to the model's development deployment—a separate, mutable slot on the same L4 instance type—using the hosting platform's development-deploy workflow. The production deployment was never touched. Each variant was warmed, then swept across the same five concurrency levels (8, 12, 16, 24, 32 concurrent users) with the same temperature-0 workload and the same shared enterprise prompt prefix.
| Item | Value |
|---|---|
| Model | Qwen 2.5 3B Instruct |
| Engine | vLLM 0.12.0 |
| Hardware | 1x L4, single replica |
| Baseline flags | --max-model-len 4096 --max-num-seqs 128 --enable-prefix-caching --enable-chunked-prefill |
| Sweep | 8/12/16/24/32 users, 60-80 requests per level |
| Admission | Hosting-platform request concurrency 32 |
Because each dev deployment was config-identical to production except for the one
changed flag, each run is a fair one-knob comparison against the same baseline
curve. This closes the honest gap from Part 1: previously only predict_concurrency
had a clean before/after; now every batching flag does.
First, Where the Latency Actually Goes
Before turning knobs, it helps to know what a single request is spending its time on. Two no-deploy measurements settled that.
Decode dominates. Splitting end-to-end latency from the engine's own histograms, across 20 sequential requests:
| Component | Mean per request |
|---|---|
| Time to first token | 35 ms (1%) |
| Prefill | 32 ms |
| Queue | 0 ms |
| Decode | 3057 ms (99%) |
The ~3.7k-token enterprise prompt is almost free to process, because it is a shared prefix and the prefix cache serves it. Ninety-nine percent of the time is decode - generating the ~113-token intent JSON one token at a time.
That makes output length the latency lever. An A/B of pretty-printed versus compact JSON, sent back to back so the cached prefix stayed byte-identical:
| Output format | p50 latency | Output tokens |
|---|---|---|
| Pretty (default) | 3243 ms | 116 |
| Compact (when honored) | 2197 ms | 77 |
When the model honored the compact instruction, it was about 32% faster - purely from generating fewer tokens. But the prompt instruction was unreliable: the model honored it 7 times out of 12, ignored it 4 times, and once produced more tokens. The lesson is not "ask for compact JSON." It is "enforce compact output at the structured-output grammar layer, where it is deterministic, not in the prompt, where it is a suggestion."
Both findings show that output generation dominated client-visible latency on this exact prefix-cached workload. They do not, by themselves, identify the hardware resource limiting decode: that would require defined compute and memory-throughput counters, kernel timings, or a controlled intervention aimed at the suspected resource.
What the Utilization Snapshot Can Establish
Under sustained 24-way load, scraping the server's own GPU and cache metrics:
| Metric | Idle | Under load |
|---|---|---|
| GPU compute utilization | (noisy) | ~0.56 -> 0.80 |
| KV cache usage | 0.9% | ~1.4% |
| Requests running | 1 | 24 |
| Requests waiting | 0 | 0 |
| Prefix-cache hit rate | - | 99.6% |
The KV cache pool sat near 1.4% full while the reported GPU activity rose and no requests queued at 24-way load. That establishes substantial KV-capacity headroom in this observation. It does not establish a natural throughput knee or prove arithmetic saturation: generic GPU activity can coexist with HBM stalls, launch overhead, or inefficient kernels. The baseline also continued scaling at the largest measured level, from 4.93 requests/sec at 24 users to 6.31 at 32, so this sweep did not locate its natural saturation point.
(One honesty note: the "idle" GPU number was noisy because the scrape caught an in-flight request. The durable inference from this snapshot is narrower: KV capacity was not binding at the observed load.)
The Four Isolated Knobs
| Knob (one changed vs baseline) | 8u | 16u | 24u | 32u | p95 at 32u |
|---|---|---|---|---|---|
| baseline | 2.14 | 4.24 | 4.93 | 6.31 | 4.5 s |
max_model_len 8192 |
2.12 | 4.24 | 4.95 | 6.34 | 4.5 s |
max_num_seqs 16 |
2.09 | 4.18 | 4.28 | 4.30 | 7.5 s |
prefix_caching off |
1.11 | 1.48 | 1.50 | 0.26 | 32.1 s |
chunked_prefill off |
2.13 | 4.25 | 4.97 | 6.34 | 4.5 s |
Throughput in req/s. The full per-level tables, including p50/p95 at every concurrency, are in the companion project.
Prefix caching was the load-bearing knob
This is the A/B that had never been run. Turning prefix caching off dropped throughput 3-4x and inflated p95 by 5-7x, collapsing to 0.26 req/s at 32 users - with the only request failure of the entire study.
The reason is the workload's shape. The enterprise prompt is a large, shared prefix. With caching on, that prefill is computed once and reused, which is exactly why decode dominates and prefill is ~1% of latency. With caching off, every request re-prefills the whole prompt, and prefill contention crushes the endpoint under concurrency.
The takeaway generalizes: prefix caching is not an incidental optimization for a system with a long shared prompt. It is the thing that makes a long shared prompt affordable at all. If your architecture puts schema, glossary, and policy in a shared prefix - as this one deliberately does - prefix caching is load-bearing infrastructure, not a nice-to-have.
Lowering max_num_seqs imposed a batch ceiling
Capping the running batch at 16 while admission still allowed 32 concurrent requests plateaued throughput at ~4.3 req/s beyond 16 users, and roughly doubled p95 (4.5 s to 7.5 s). The extra requests could not join the batch, so they queued.
max_num_seqs capped the number of running sequences in this configuration, so the imposed
ceiling appeared once offered concurrency exceeded 16. The baseline's cap of 128 was not
binding within the tested range; external admission concurrency of 32 bounded the
largest offered level. Modern schedulers also enforce a per-iteration token budget, so
max_num_seqs is not a complete definition of the realized batch. Treat it as one capacity
control to sweep against the latency SLO, not a rule that must always equal admission.
max_model_len 4096 to 8192 had no measurable cost here
Doubling the context window produced an identical throughput and latency curve. On this L4, with KV usage near 1.4%, there was enormous headroom, so a larger per-sequence KV footprint cost nothing at these batch sizes.
The bounded finding is that this context-window change had no measurable throughput or latency cost in the tested range, where KV occupancy stayed low. Context-window sizing here is capacity planning: size it for the prompt plus valid output and headroom, then repeat the sweep as lengths or concurrency change. This experiment rules out a KV-capacity limit in the observed runs; it does not identify what resource would bind next.
Chunked prefill did nothing measurable - and that is a result
Turning chunked prefill off produced the same curve as baseline. That is a null result, and it is worth stating plainly rather than quietly dropping.
Chunked prefill earns its keep by interleaving long prefills with other requests' decode, so a big prompt does not stall the batch. On this workload there is nothing to interleave: prefill is tiny (prefix-cached) and outputs are short. The flag is correct to leave on as insurance, but on this traffic shape it is not what makes the endpoint fast. It would start to matter with long unique prefills competing against decode - a different workload than this one.
What This Says About Production Architecture
The measured knobs turn the roadmap's open design questions into defaults with evidence behind them.
Context sizing is capacity planning. Size max_model_len from the observed input plus
output length plus headroom, not from a desire to maximize it. Raise it with a before/after
sweep, and watch KV usage, throughput, and tail latency—not just whether it boots.
There is no universal best concurrency. The same endpoint supports different valid operating policies, and the right one depends on what you are optimizing:
| Mode | Optimize for | Behavior on this endpoint |
|---|---|---|
| latency-first | p95 below target | choose the highest tested load that preserves the p95 target |
| cost-first | lowest cost per query | compare cost per SLO-compliant request across the measured range |
| throughput-first | maximum completed work | extend the baseline sweep until a real plateau or SLO boundary appears |
| quality-first | semantic pass rate | route hard questions to a stronger model or a clarifier |
The open frontier is multi-replica. Everything here is single-replica, so it isolates engine behavior, not routing. The next questions are genuinely harder and deliberately out of scope for this measurement: round-robin routing can create hot workers when requests differ in token cost; routing for load balance can destroy the prefix-cache reuse that this article just showed is worth 3-4x; and autoscaling has to react to SLO pressure before a ~2-minute cold start has already broken the SLO. Those are design problems that a single replica cannot answer, and I would rather name them than pretend one L4 settled them.
Caveats
- Single replica, single L4, one model. These are engine-behavior results, not a routing or autoscaling study.
- The workload has a large shared prefix and short outputs. The prefix-caching and chunked-prefill results are specific to that shape; a long-unique-prefix workload would rank chunked prefill very differently.
- Latency is client-observed (network + client queue + server queue + inference). The TTFT/decode split uses the server's own histograms and is labeled as such.
- Sweep levels use 60-80 requests each. That is enough to show the imposed
max_num_seqs=16ceiling, but the baseline was still scaling at the largest level and its tail percentiles carry ordinary small-sample noise. - The dev-slot comparison assumes the development and production deployments behave identically given identical config on the same instance type, which is the intended contract but not separately proven here.
Reproducibility
The companion project contains the sweep driver, the metrics scrapers, the config variants, and every raw sweep JSON:
results/runs/2026-07-24_a16_redeploy_knob_sweeps.md- the four isolated knobs.results/runs/2026-07-24_a16_no_redeploy_sweeps.md- the decode split, the compact-output A/B, and the GPU-under-load scrape.results/METRICS-AND-KNOBS.md- the full knob -> observable -> caution reference, now with a clean before/after for every batching flag.scripts/loadtest.py,scripts/ttft_decode_split.py,scripts/gpu_under_load.py,scripts/ab_json_format.py,scripts/plot_a16_knob_sweeps.py.
The one-line summary I would give a teammate: on this generation-dominated endpoint with a long shared prompt, prefix caching was the load-bearing knob, a deliberately lowered sequence cap imposed a ceiling, the tested context-window increase had headroom, and the baseline's natural saturation point remains beyond the measured sweep.