Latency, throughput and token rates

LLM inference latency metrics describe more than whether a system is “fast.” A streamed response has a startup delay and a generation rhythm, while the serving system has a total work rate. Those measurements are related to prefill and decode, but none identifies one unique hardware limit by itself.

Getting the vocabulary right—time to first token, inter-token latency, throughput, and goodput—lets you write a meaningful SLO, compare benchmarks, and choose the next measurement when something is slow.

What you will understand by the end

• The core latency metrics—TTFT, ITL/TPOT, and end-to-end latency—and their precise measurement boundaries.
• The difference between latency for one request and throughput for the system.
• Why each metric suggests where to investigate first but does not prove a bottleneck.
• How to report central tendency, tail behavior, sample size, and workload conditions.

The startup delay and generation rhythm

For output arrival times t₁ ... tN, a client can observe:

  request sent
     │
     │  ⏱ TTFT — time until the first output arrives
     ▼
   output₁ · output₂ · output₃ · ... · outputN
             └ gap ┘  one inter-output interval

  end-to-end latency = TTFT + the sum of the N−1 intervals

• TTFT—time to first token. The time from submitting a request until receiving the first token or visible text chunk. Client-observed TTFT can include tokenization, request handling, network transit, queueing, scheduling, prefill, and delivery of the first output.
• Inter-token latency (ITL). An individual interval between consecutive token arrivals. Some tools instead use “average ITL” for the request-level average of those intervals.
• TPOT—time per output token. Commonly the request-level average generation interval, excluding the first token. Definitions differ across tools, so state the formula used.
• End-to-end latency. The time from request submission until the complete response is received.

If a request produces N > 1 outputs and TPOT is the average of its N−1 generation intervals, the exact relationship at that measurement boundary is:

end-to-end latency = TTFT + (N−1) × TPOT

At the client boundary, a streaming API may buffer several model tokens into one response chunk. In that case, client timestamps measure time to first chunk and inter-chunk gaps—not necessarily engine token-to-token latency. Define whether “output” means an engine token or a user-visible response chunk.

From symptom to diagnosis

Measured symptom	Investigate first	Measurements that help distinguish causes
TTFT rises with prompt length at low load	prefill work, kernels, prefix-cache behavior	server queue time, prefill execution time, prompt tokens, cache hit rate
TTFT rises with offered load but not prompt length	admission, queueing, capacity	arrival rate, outstanding requests, queue depth/time, running batch
Generation gaps grow with context	KV traffic, attention work	resident tokens, KV bytes, per-iteration time, memory and compute counters
Generation gaps spike when new requests arrive	prefill/decode interference, scheduling	iteration traces, prefill admissions, running token budget, gap distribution
Generation is slow even at low batch	weights, KV traffic, kernels, launch or communication overhead	achieved bandwidth, tensor activity, kernel trace, parallel communication

Low-batch dense decode is often bandwidth-sensitive, but context length, batch composition, kernels, MoE routing, parallelism, communication, and scheduler interference can change the limit. Likewise, slow TTFT is not proof that prefill execution dominates: queueing or the network may be larger.

Latency, throughput, and goodput

• Latency describes one request: TTFT, generation intervals or TPOT, and end-to-end latency.
• Throughput describes total completed work per unit time: for example output tokens/s or requests/s for a replica or whole system.
• Request goodput is the number of completed requests per second that satisfy the defined TTFT and TPOT SLOs. Other goodput definitions are possible, so state the unit and acceptance rule. Raw throughput can look healthy even when many requests miss the SLOs.

request goodput = SLO-compliant completed requests ÷ benchmark duration

Concurrency and engine batch size are different. Concurrency is outstanding client work; the engine's running batch may contain only a subset of those requests. Excess requests may wait in a queue.

Increasing effective batch size often raises aggregate throughput until saturation by reusing loaded weights, amortizing fixed overhead, and improving hardware utilization. It can also increase per-request latency through longer iterations, interference, or queueing, especially near and beyond the saturation knee. The result depends on offered load, continuous versus static batching, token budgets, and scheduler policy. Chunked prefill and other scheduling changes can shift the frontier, not merely move the system along one fixed tradeoff.

Tokens/sec also needs care across models. Different tokenizers divide the same text differently, and systems may produce outputs of different length and quality. Compare the same workload and report task quality or useful completions per second alongside token rate.

Typical and tail latency

Averages alone can hide poor tail behavior because serving latency is often skewed under bursty or saturated traffic. The mean is still useful for expected user time, resource and cost models, and distribution comparisons. Report central tendency together with the percentile required by the SLO:

• Mean—the arithmetic average; sensitive to extreme observations.
• p50 (median)—the threshold at or below which about half of observations fall.
• p95—the threshold at or below which about 95% fall; about 5% are slower.
• p99—the threshold at or below which about 99% fall; about 1% are slower.

Always name the population. The p99 of pooled token gaps, the p99 of each request's average TPOT, and the p99 of each request's worst gap answer different questions. Pooling intervals also gives longer responses more weight. For a request-level SLO, a percentile across request-level TPOT values is usually easier to interpret.

Percentiles need sample context. Report the request count and test duration; with only tens of requests, an empirical p99 is controlled by roughly one extreme observation and is not a precise tail estimate.

Reproducible benchmark context

A number without its workload and measurement boundary is not reproducible. Record:

• metric formulas and whether timing is client-observed or server-internal;
• streaming behavior, network boundary, and whether arrivals are tokens or chunks;
• input and output length distributions and tokenizer;
• concurrency or arrival-rate process, plus engine batch and token limits;
• model, numeric format, quantization, sampling settings, and EOS behavior;
• serving engine and version, hardware, replica count, and parallelism topology;
• cache state and prefix-cache hit rate;
• warm-up, run duration, request count, and repeated-trial strategy.

Then plot latency and throughput—or goodput—across a load sweep. A plateau with rising queue time establishes saturation, but attributing it to compute, bandwidth, capacity, kernels, or communication requires corresponding measurements and preferably a controlled change.

Mental model

An LLM response has a startup delay and a generation rhythm. TTFT and TPOT describe user-visible symptoms; throughput describes total work; goodput describes work completed within the SLO. Each narrows the investigation, but the workload, measurement boundary, and controlled evidence determine the cause.

Common mistakes

• Turning a phase association into a diagnosis. TTFT does not prove prefill is slow, and poor ITL does not prove HBM bandwidth is saturated.
• Treating concurrency as batch size. Outstanding requests can be queued rather than in the engine's running batch.
• Quoting an undefined token rate. Counted tokens, scope, workload, and latency constraints determine what the number means.
• Reporting only a mean or only a percentile. Report both, plus sample count and duration.
• Mixing token gaps with response-chunk gaps. The application boundary may buffer output.
• Claiming a bottleneck from a knee alone. Saturation is a symptom; resource counters and controlled interventions establish its cause.

Practical guidance

1. Define TTFT, TPOT/ITL, end-to-end latency, throughput, and goodput at one explicit measurement boundary.
2. Report mean or median with the SLO percentile, request count, duration, and repeated-run variability.
3. Characterize traffic, sequence lengths, engine limits, model configuration, hardware, software, cache state, and sampling behavior.
4. Use TTFT and generation behavior to form phase-level hypotheses, then inspect queueing, traces, counters, and communication before naming a bottleneck.
5. Change one workload or resource dimension and test whether the metric moves as predicted.
6. Choose an operating point by goodput or latency-constrained capacity, not peak raw throughput alone.

Summary

• TTFT measures startup delay; individual ITLs describe generation gaps; request-level TPOT commonly averages those gaps. Tool definitions can differ.
• Client measurements include more than model execution and may observe chunks rather than engine tokens.
• Latency is per request, throughput is total work, and goodput is work meeting the SLO.
• Batching can improve aggregate throughput, but its latency effect depends on load, scheduling, and the operating regime.
• Report typical and tail behavior with workload, sample, and measurement context; use controlled evidence before assigning a cause.

Knowledge check

Users report a long pause before output starts, but generation is smooth. What should you measure next, and what fixes might follow?

The symptom is poor TTFT, while the generation metric appears healthy. Separate client network and request-processing time, server queue time, and prefill execution time. If queueing dominates, reduce load or admission delay, change scheduling, or add capacity. If prefill dominates, shorten or prefix-cache prompts or improve prefill execution. A smaller model may improve both TTFT and TPOT when its quality tradeoff is acceptable; the symptom alone does not establish that it is the best fix.

A benchmark advertises “3,000 tokens/sec.” What must you learn before inferring user-visible speed?

Ask whether it counts input, output, or combined tokens and whether its scope is per user, GPU, replica, or whole system. You also need TTFT and TPOT definitions, concurrency or arrival rate, input/output lengths, engine batch limits, latency constraints, and the model, hardware, and software configuration. The headline alone does not reveal a user's experience.

Why does a throughput plateau with rising p95 latency not prove the system is compute-bound?

It establishes that offered load has reached a saturation region and work is waiting longer. Compute, memory bandwidth, capacity, kernels, communication, or another serialized resource could produce that symptom. Inspect queueing and engine traces, resource counters, and a controlled change to the suspected limit before making a causal claim.

Primary sources

• NVIDIA NIM benchmarking metrics — TTFT, end-to-end latency, ITL/TPOT, and system versus per-user TPS definitions.
• NVIDIA NIM benchmarking parameters — concurrency, engine batch size, request rate, sequence lengths, EOS, and sampling controls.
• Sarathi-Serve — prefill interference, chunked prefill, and latency-constrained serving capacity.
• DistServe — goodput under TTFT and TPOT SLOs.

Related chapters

• Prefill and decode — phase behavior and regime-dependent bottlenecks
• Batching and concurrency — running batches, queues, and scheduling
• The KV cache and context growth — capacity and traffic as context grows