Part 7 — Inference Performance and Economics

TTFT, inter-token latency and time to last token

Latency metrics·Serving·6 min read

The generation segment has its own three-number vocabulary — time to first token, inter-token latency, and time to last token — and each maps to a different phase, a different cause, and a different user experience. This is the precise reference for the metrics every serving benchmark and SLO is written in.

Within the end-to-end sum, the generation segment deserves its own precise vocabulary, because "generation latency" hides three distinct experiences with three distinct causes. Time to first token (TTFT) is the opening pause; inter-token latency (ITL) is the streaming rhythm; time to last token (TTLT) is when the whole answer is done. Every serving benchmark, SLO, and vendor claim is written in these three, and confusing them — or collapsing them into one "speed" — is how latency conversations go wrong.

What you will understand by the end

  • The precise definitions of TTFT, ITL (TPOT), and TTLT, and how they relate.
  • Which inference phase and cause each maps to.
  • Why a system can be great on one and bad on another.
  • How answer length ties them together.

The three numbers

Building on fund-latency, stated precisely:

  • TTFT — time to first token. From request accepted to the first output token. Dominated by prefill (processing the whole prompt) plus any queueing. Long prompts and busy servers hurt it.
  • ITL — inter-token latency (a.k.a. TPOT, time per output token). The gap between successive streamed tokens once generation starts. This is the decode speed, governed by memory bandwidth and how loaded the batch is.
  • TTLT — time to last token. From request to the final token — the whole answer complete. Approximately TTFT + (N−1) × ITL for an N-token answer.
   request
      │  ⏱ TTFT ──────────────┐ (prefill + queue)
      ▼                        ▼
    tok₁ · tok₂ · tok₃ · ... · tok_N
          └ ITL ┘   ↑ inter-token latency (decode), one gap per token
      │                                    │
      └──────────── TTLT ──────────────────┘  ≈ TTFT + (N−1)·ITL
Key idea

TTFT is "how long until it starts," ITL is "how fast it streams," and TTLT is "when it's fully done." They map to different phases (prefill vs decode) with different cures, and TTLT is a function of both plus answer length. A single "latency" number picks one of these and hides the rest.

Which one matters depends on the experience

The metric you optimise is set by how the answer is consumed:

  • Streaming, read-as-it-generates (chat, assistants): TTFT dominates the felt responsiveness — users tolerate a steady stream once it starts, but hate the opening pause. Keep TTFT low.
  • Wait-for-the-whole-answer (a structured result a program consumes, a batch job): TTLT is what matters — the user or system can't act until the full answer lands, so total time rules.
  • Long generations (reasoning, long documents): ITL dominates TTLT, because (N−1) × ITL is most of the time — so per-token speed is the lever.
Watch out

A vendor "tokens/sec" or "fast" claim usually collapses these. A system can have excellent TTFT and mediocre ITL (snappy to start, slow to finish) or the reverse, and which is "fast" depends entirely on your use case. Always ask for TTFT and ITL separately (and the answer length assumed) before comparing — one number is not comparable across systems or use cases.

Answer length is the hidden variable

TTLT depends on how many tokens you generate, so output length is a latency lever you control. Two systems with identical TTFT and ITL differ in TTLT purely by how long their answers are — and a prompt that elicits a shorter, equally-correct answer is a real latency (and cost) improvement that no model change is needed for. When you compare token latencies, hold answer length fixed, or you're measuring verbosity, not speed.

Observed evidence

This project's structured-output task returns a compact QueryIntent, not prose — a deliberately short output, which keeps TTLT close to TTFT and makes the workload prefill/first-token dominated rather than decode-dominated. That's a design choice with a latency consequence: asking the model for a small intent instead of a long SQL string minimises the `(N−1)·ITL` term. Output shape is a latency decision, not just a correctness one. See the compact structured output →

Mental model

TTFT (prefill + queue — the opening pause), ITL/TPOT (decode — the streaming rhythm), and TTLT ≈ TTFT + (N−1)·ITL (the whole answer done). Different phases, different cures; which one matters depends on whether the answer streams or is awaited, and TTLT scales with answer length — a lever you control. Never compare a single "speed" number; separate the three and fix answer length.

Common mistakes

  • Quoting one "generation speed." TTFT and ITL are different metrics with different causes; collapsing them prevents diagnosis.
  • Optimising the wrong one. Shrinking the model to fix a slow TTFT (a prefill/queue problem) misses the cause.
  • Comparing token latencies at different answer lengths. You're measuring verbosity, not speed; hold N fixed.
  • Ignoring output length as a lever. A shorter equally-correct answer cuts TTLT and cost with no model change.

Practical guidance

  • Report and gate on TTFT and ITL separately (plus TTLT for awaited answers), always with the answer length assumed.
  • Choose the target metric by consumption: streaming → TTFT; awaited/whole-answer → TTLT; long generations → ITL.
  • Treat output length as a first-class latency (and cost) lever — prompt for concise answers, cap max tokens.
  • Diagnose by phase: slow TTFT → prompt size, prefix caching, queue; slow ITL → model size, quantization, bandwidth, batch load.

Summary

  • TTFT (prefill + queue), ITL/TPOT (decode), and TTLT (≈ TTFT + (N−1)·ITL) are the three generation metrics — different phases, different cures.
  • Which matters depends on the experience: TTFT for streaming, TTLT for awaited answers, ITL for long generations.
  • Answer length ties them together and is a latency/cost lever you control — hold it fixed when comparing.
  • These are the precise metrics every benchmark and SLO is written in; never collapse them into one "speed."

Knowledge check

A chat assistant feels sluggish: users wait a long time before any text appears, but once it starts it streams fine. Which metric is bad, which phase, and what are two fixes — and which fix would be wrong?

The bad metric is TTFT (long opening pause); the streaming being fine means ITL is okay. TTFT is governed by prefill plus queueing, so the right fixes target those: shorten or prefix-cache the prompt, and reduce queueing (more capacity / batch tuning). The wrong fix would be shrinking or swapping the model to speed up generation — that improves ITL, which isn't the problem, and does little for the prefill/queue time that's actually causing the sluggish start.

Two systems both report ITL of 20 ms/token and TTFT of 200 ms, but one "feels much slower" for a summarization feature. What likely explains it?

Answer length. TTLT ≈ TTFT + (N−1)·ITL, so with identical TTFT and ITL the total time is driven by how many tokens each system generates. If one produces much longer summaries (say 400 tokens vs 150), its TTLT is far higher purely from the (N−1)·ITL term — it feels slower because the user waits for a longer answer, not because per-token speed differs. The fix is output length, not the model: prompt for more concise summaries or cap max tokens. It also shows why you must hold answer length fixed when comparing token latencies.

Related chapters