Latency, throughput and token rates

"Is it fast?" is the wrong question, because an LLM has at least two different speeds that a user experiences at different moments, and they are governed by different hardware limits. Getting the vocabulary right — time to first token, inter-token latency, throughput — is what lets you write a meaningful SLO, read a benchmark, and know which knob to turn when something is slow. Better still, each metric maps cleanly onto the two phases you already know, so a number points straight at a cause.

What you will understand by the end

• The core latency metrics — TTFT, ITL/TPOT, end-to-end — and what each one feels like.
• The difference between latency (one request's experience) and throughput (the system's total rate).
• Why each metric is governed by a different phase and hardware limit.
• Why you always report tail latency (p95/p99), never just the average.

The two speeds a user feels

Streaming a response has a distinct rhythm: a pause, then words appearing one after another. Those are two separate metrics:

  request sent
     │
     │  ⏱ TTFT (time to first token) ── the pause before anything appears
     ▼
   token1 · token2 · token3 · ... · tokenN
            └─ ITL ─┘  inter-token latency: the gap between streamed tokens
     │
     ▼  end-to-end latency = TTFT + (N−1) × ITL  ── total time to the full answer

• TTFT — time to first token. How long until the first word appears. This is dominated by prefill (reading and processing the whole prompt) plus any queueing before the request is scheduled. Long prompts and busy queues hurt TTFT.
• ITL — inter-token latency (a.k.a. TPOT, time per output token). The gap between successive streamed tokens once generation starts. This is the decode speed, governed by HBM bandwidth and how loaded the batch is.
• End-to-end latency. The whole answer's time, ≈ TTFT + (N−1) × ITL. For a long answer ITL dominates; for a short answer TTFT dominates.

Latency versus throughput

Latency is about one request; throughput is about the whole system:

• Latency — how long your request takes (TTFT, ITL, end-to-end). What a single user feels.
• Throughput — how much the system does per unit time, usually tokens/sec aggregate or requests/sec. What your cost-per-token and capacity depend on.

They pull in opposite directions, and batching is the reason:

   larger batch ─┐
                 ├─►  throughput ↑   (tokens/sec across all users)
                 └─►  per-request latency ↑   (each user waits a bit more)

Why you report the tail, not the average

Averages lie about user experience because latency distributions are skewed — a few requests (long prompts, unlucky queueing, a big batch moment) take far longer than the median. The users who churn are the ones on the slow tail, so SLOs are written on percentiles:

• p50 (median) — the typical case; hides the bad ones.
• p95 / p99 — the slow tail; 1-in-20 and 1-in-100 requests are at least this slow.

A system with a great p50 and a terrible p99 feels unreliable, because everyone eventually hits the tail. "Fast on average" is not a serving guarantee.

Mental model

An LLM has two speeds: TTFT (the prefill pause) and ITL (the decode drip), and end-to-end is their sum. Latency is one request's experience; throughput is the system's total rate; batching trades one for the other. Judge all of it on the tail, not the mean.

Common mistakes

• Quoting one "speed." TTFT and ITL are different metrics with different causes; collapse them and you can't diagnose anything.
• Comparing aggregate throughput to single-stream latency. They measure different things at different batch sizes — apples to oranges.
• Reporting averages. The tail is where users suffer and where SLOs live; p99 ≫ mean for LLM traffic.
• Optimising the wrong phase. Shrinking the model to fix a slow TTFT misses the point if the real cost was queueing or a giant prompt.

Practical guidance

• Write SLOs as p95/p99 of TTFT and ITL (or end-to-end for a target answer length), plus a throughput target — never a single average.
• Diagnose by phase: slow TTFT → prompt length, prefix caching, queueing, capacity; slow ITL → model size, quantization, bandwidth, batch load.
• When you benchmark, always record the batch size and concurrency alongside the numbers, or they are not reproducible or comparable.
• Track latency and throughput together; improving one usually moves the other, and the right operating point is the knee under your SLO.

Summary

• TTFT (prefill + queueing), ITL/TPOT (decode bandwidth), and end-to-end (their sum) are the latency metrics; each maps to a phase and has its own cure.
• Latency is per-request, throughput is system-wide (tokens/sec, req/sec), and batching trades one for the other.
• Report and gate on the tail (p95/p99), because skewed latency distributions make the average a poor guide to real experience.

Knowledge check

Users complain about a long pause before the answer starts, but once it starts it streams fine. Which metric is bad, which phase is responsible, and name two fixes?

The bad metric is TTFT (time to first token); the streaming being fine means ITL is okay. TTFT is governed by prefill plus queueing, so fixes target those: shorten or prefix-cache the prompt, or add capacity/reduce load to cut queue time — not shrinking the model, which would help ITL you don't have a problem with.

A benchmark advertises "3,000 tokens/sec." Why is that not enough to know how fast a user's request will feel?

Because 3,000 tokens/sec is almost certainly aggregate throughput across a large batch, not the per-request ITL a single user experiences. High aggregate throughput and slow single-stream latency coexist happily. You'd need the per-request TTFT and ITL, and the batch size/concurrency the number was measured at, to know what a user feels.

Related chapters

• Prefill and decode — the two phases TTFT and ITL map onto
• Batching and concurrency — why throughput and latency trade off
• The KV cache and context growth — the memory limit behind the throughput knee