LLM batching and concurrency

For throughput-oriented GPU serving, batching is usually the main way to amortize model-weight traffic and runtime overhead across generated tokens. Continuous batching, also called in-flight batching or iteration-level batching, lets the engine reconsider that work at each scheduling boundary. It is less valuable when requests do not overlap or when strict single-request latency matters more than aggregate throughput. This chapter separates request concurrency from the work an inference engine actually batches, then shows how irregular traffic becomes scheduled GPU work.

What you will understand by the end

• Why dense autoregressive decode often benefits strongly from batching—and where that first-order heuristic breaks.
• The difference between request concurrency, running sequences, iteration batches, and token budgets.
• How continuous batching schedules prefill and decode work under resource and policy limits.
• Why throughput gains and their effects on TTFT, ITL, latency, and memory are workload-dependent.
• How to identify an operating region without guessing at the limiting resource.

Why decode often benefits from batching

For a conventional dense decoder in a common small-batch regime, each decode step uses relatively little arithmetic per byte of model weights moved from HBM. A batch lets a forward pass apply much of that weight traffic to several tokens instead of one, increasing arithmetic intensity and using more of the GPU's parallel capacity.

  BATCH 1                          BATCH 8
  model weights ──► 1 token        model weights ──► up to 8 tokens
  little work per weight byte      more work per weight byte

This is amortization, not free or linear scaling. Arithmetic work, activations, sampling, and KV-cache traffic grow with the batch. Attention work depends on each sequence's context. Kernel efficiency changes nonlinearly, and the next constraint could be compute throughput, HBM traffic, CPU or launch overhead, communication, scheduler policy, or available demand. Mixture-of-experts routing can also change which weights each token uses.

Concurrency is not batch size

“Batch size” becomes ambiguous once requests enter and leave continuously. Keep these quantities separate:

Concept	Meaning
Request concurrency	Requests simultaneously offered to or outstanding at the server
Admitted requests	Requests accepted by the serving layer but not necessarily running
Running sequences	Requests admitted into the engine's active set
Iteration batch	Prefill and decode work selected for one scheduling iteration
Token budget	Maximum scheduled prefill and decode tokens in that iteration
Queue depth	Requests waiting outside or inside the engine for service
Batch cap	A configured sequence or token limit, not necessarily the realized batch

For example, 64 outstanding requests do not imply an iteration batch of 64. Some may wait at the gateway, some may wait for engine admission, and the scheduler may select only work that fits its sequence, token, memory, priority, and latency budgets.

max_num_seqs versus max_num_batched_tokens

In vLLM, max_num_seqs limits how many sequences can be scheduled in one iteration, while max_num_batched_tokens limits the total number of tokens scheduled in that iteration. One hundred short decode steps and one long prefill stress those limits differently: the first may reach the sequence cap, while the second may consume the token budget alone or be split into chunks. Neither setting equals offered request concurrency, and neither guarantees that the engine will realize its configured maximum on every iteration. In current vLLM, max_num_scheduled_tokens normally equals max_num_batched_tokens, but some paths that can append tokens—such as speculative decoding—may apply a smaller effective scheduling limit.

Consider an iteration with max_num_seqs=8 and max_num_batched_tokens=256. Six running decode sequences need one token each. The scheduler can use the remaining 250-token budget for a chunk of one waiting prefill: 6 decode tokens + 250 prefill tokens = 256 scheduled tokens across seven sequences. One sequence slot is still nominally free, but another token cannot enter this iteration because the token budget is full.

Static versus continuous batching

The naive approach is static batching: gather several requests and keep the group fixed until every request finishes. Because output lengths differ, completed members leave wasted slots while shorter arrivals wait for the longest member.

Continuous batching—also called in-flight or iteration-level batching—reconsiders the running work at scheduling boundaries:

At each iteration, a modern runtime selects decode tokens and, depending on policy, prefill tokens or chunks. The selection is bounded by limits such as maximum running sequences, maximum scheduled tokens, available KV blocks, priority or FCFS policy, and preemption or fairness rules. Finished requests leave the running set. Waiting requests may be admitted at a later iteration when those budgets permit; arrival does not guarantee immediate entry.

This is broader than “one decode step for every active request.” Orca introduced iteration-level scheduling with selective batching. Current vLLM scheduling distinguishes max_num_seqs from max_num_batched_tokens; with chunked prefill, decode work can be prioritized while remaining token budget admits prefill chunks. TensorRT-LLM can likewise schedule context-phase and generation-phase sequences together.

Throughput, latency, and memory do not move in lockstep

Larger realized iteration batches often improve aggregate throughput until another resource or policy becomes binding. Their latency effect is not a universal upward arrow:

• TTFT includes queueing, admission, and prefill. More concurrency can increase queueing, while a larger prefill token budget can sometimes improve prompt progress and TTFT.
• ITL/TPOT measures generation spacing. Large prefills mixed with decode can disrupt it; decode-prioritizing or smaller token budgets can protect it.
• End-to-end latency combines startup and generation, so input/output lengths and scheduling both matter.
• Tail latency is especially sensitive to queue growth, request-size variance, preemption, and scheduling policy.

  more offered concurrency
          │
          ├── may create larger realized iteration batches
          ├── may wait at admission or in a queue
          └── changes TTFT and ITL differently as scheduling budgets bind

  aggregate throughput rises until a resource, policy, or demand limit binds

KV capacity is an important part of this system, but it is not the whole memory budget. Weights, runtime and kernel workspaces, temporary activations, CUDA graph buffers, logits and sampling state, communication buffers, fragmentation, and optional multimodal or speculative state also consume memory. After those costs, KV is often the main workload-dependent limit on the number and context length of concurrent sequences.

Paged KV allocation reduces waste from reserved-but-unused capacity and fragmentation; it does not make KV the only ceiling. Other ceilings include model-weight or attention bandwidth, arithmetic throughput, CPU scheduling and kernel launches, interconnect collectives, prefill interference, runtime kernels, admission limits, and simply not having enough overlapping requests.

What the L4 experiment actually established

Generic GPU activity plus low KV occupancy is not a roofline diagnosis. Low occupancy can rule out a KV-capacity ceiling for the observed workload. Establishing a compute or bandwidth bottleneck requires suitable counters and controlled evidence: achieved FLOPs or tensor-core activity, DRAM throughput, kernel timings, communication traces, or an intervention that changes the suspected resource and moves performance as predicted.

Mental model

In dense autoregressive decode, batching often amortizes model-weight traffic and runtime overhead across multiple generated tokens. Gains are sublinear and workload-dependent because compute, KV and attention traffic, prefill work, scheduler overhead, and memory all change with the running workload. Continuous batching lets requests enter and leave at scheduling boundaries, subject to sequence, token, memory, priority, and latency budgets.

Common mistakes

• Treating request concurrency as the running batch. Outstanding requests may be queued outside or inside the engine.
• Dismissing batch-1 benchmarks. They are valuable for minimum decode latency, single-user interaction, overhead, edge deployments, and hardware comparisons; they say little about loaded production throughput by themselves.
• Calling batching free or mandatory. It needs overlapping demand and consumes compute, memory, and scheduling time. Batch 1 can be rational for low traffic or strict latency.
• Equating max_num_seqs with max batch. Sequence and scheduled-token limits constrain different dimensions, and neither guarantees the realized iteration batch.
• Treating KV capacity as the only ceiling. Measure memory and execution resources rather than inferring one from a config value.
• Naming a bottleneck from utilization alone. Activity and occupancy metrics need defined semantics, relevant counters, and controlled tests.

Practical guidance

1. Generate overlapping offered load representative of production; record request concurrency, running sequences, iteration composition, scheduled tokens, and queue depth.
2. Sweep sequence and token budgets separately. Report TTFT, ITL/TPOT, end-to-end and tail latency alongside request and token throughput.
3. Include prompt/output-length distributions, cache state, model/runtime versions, hardware, scheduling policy, sample count, and run-to-run variation.
4. Diagnose the first binding resource with engine traces, profiler counters, and a one-variable intervention—not with KV occupancy or generic GPU activity alone.
5. Choose the operating region from arrival rate, SLO goodput, cost, queue growth, and resilience headroom. Depending on the goal, that may be below saturation, near a plateau, or at a lower cap that protects ITL.
6. Use admission control and autoscaling before an unbounded queue turns offered load into tail-latency failure.

Summary

• Batching often improves dense-model decode economics by amortizing weight traffic and overhead, but gains are sublinear and conditional.
• Request concurrency, running sequences, iteration batches, and token budgets are distinct.
• Continuous batching schedules prefill and decode work at iteration boundaries; admission depends on resource and policy budgets rather than being immediate.
• TTFT and ITL can respond differently to the same scheduling change, and KV capacity is one of several possible limits.
• The L4 sweep measured an imposed max_num_seqs=16 ceiling and KV headroom; it did not find the baseline's natural knee or establish a compute bottleneck.

Knowledge check

Your server has many concurrent requests waiting, but only one sequence is running at a time. GPU throughput remains low. What should you investigate?

Check whether the client or gateway is serializing requests, admission concurrency is one, max_num_seqs is too low, the scheduled-token budget prevents useful co-scheduling, or the runtime/integration is not using iteration-level batching. Confirm with offered concurrency, engine running/waiting metrics, and iteration traces rather than GPU utilization alone.

Why can increasing a scheduler's token budget improve TTFT but hurt ITL?

A larger budget can schedule more prefill work per iteration, advancing waiting prompts and reducing time to first token. That same prefill work can occupy iterations alongside or ahead of decode work, widening gaps between generated tokens. The outcome depends on scheduler policy and workload, so measure both metrics rather than calling the change simply “faster.”

Why can low KV-cache occupancy rule out a capacity ceiling but not prove that the GPU is compute-bound?

Low occupancy shows that the measured workload had unused KV capacity. It says nothing directly about arithmetic throughput, HBM traffic, kernel efficiency, launch overhead, or communication. A compute-bound diagnosis needs relevant hardware counters, kernel profiling, or a controlled resource change whose result matches the hypothesis.

Primary sources

• Orca, OSDI 2022 — iteration-level scheduling and selective batching.
• PagedAttention and vLLM, SOSP 2023 — paged KV management, fragmentation, and serving capacity.
• Sarathi-Serve — prefill/decode interference, chunked prefills, and latency-constrained throughput.
• vLLM optimization and tuning — scheduled-token budgets and the TTFT/ITL trade-off.
• vLLM scheduler configuration — sequence, token, priority, and scheduling controls.
• TensorRT-LLM in-flight batching — mixed context and generation work.
• TensorRT-LLM memory usage — KV admission and non-KV runtime allocations.

Related chapters

• Prefill and decode — phase behavior and regime-dependent bottlenecks
• The KV cache and context growth — physical capacity and traffic as context grows
• Latency, throughput and token rates — metric definitions and causal diagnosis
• Inference runtimes — where scheduling and memory management live