Part 7 — Inference Performance and Economics

End-to-end latency

Latency metrics·Serving·6 min read

The number a user actually feels is not the model's latency — it's the whole path, from their request leaving to your answer arriving. This chapter decomposes end-to-end latency into its parts, so when it's too slow you know which segment to fix instead of blaming 'the model'.

Latency, throughput and token rates introduced the two speeds a user feels; this part goes operational. It opens with the number that ultimately matters — end-to-end latency, the total time from the user's request leaving their device to your complete answer arriving — and its most useful property: it's a sum of segments, only one of which is the model. Decompose it and a "the app is slow" complaint becomes a specific, fixable segment; leave it as one number and every fix is a guess.

What you will understand by the end

  • What end-to-end latency includes beyond model time.
  • The segments it decomposes into, and which are usually the culprits.
  • Why the model is often not the slowest part.
  • How this frames the more precise token-level metrics that follow.

End-to-end is a sum of segments

The time a user waits is everything between their request and your response, and only a slice of it is the model generating tokens:

   user request ─▶ [ network ] ─▶ [ queue ] ─▶ [ prefill / TTFT ] ─▶ [ decode / generation ] ─▶ [ post-process ] ─▶ response
                     to server      waiting       read the prompt       emit the tokens            parse, exec,        to user
                                    to be           (first token)         (rest of answer)          format
                                    scheduled
   └──────────────────────────── end-to-end latency = the sum of all of these ────────────────────────────┘
  • Network — request/response transit, TLS, any gateway hops.
  • Queue — waiting to be scheduled when the server is busy (queueing).
  • Prefill → TTFT — reading the prompt to produce the first token.
  • Decode → generation — emitting the rest of the answer, token by token.
  • Post-process — parsing, validation, execution, formatting, and any downstream calls.
Key idea

End-to-end latency is a sum, and the model's generation is only one term in it. A slow response can come from the network, a busy queue, a huge prompt, a long answer, or heavy post-processing — so "it's slow" is not a diagnosis until you know which segment dominates. Measure the parts, not just the total.

The model is often not the bottleneck

It's tempting to equate "slow response" with "slow model," but the segments around it frequently dominate. A busy server can spend more time queueing than generating; a retrieval step or a downstream API call in post-processing can outweigh the model; a bloated prompt inflates prefill; network alone can be significant for distant users. Attributing the whole end-to-end number to the model leads you to swap models when the real fix was smaller batches, a shorter prompt, or a faster downstream call.

Watch out

Optimising the wrong segment is the classic waste: shrinking or swapping the model to fix a latency problem that actually lives in queueing (fix with capacity/batch tuning) or post-processing (fix the downstream call). You can only avoid it by measuring the segments — the same stage-level discipline as failure analysis, applied to time instead of correctness.

Toward the precise metrics

End-to-end is the honest top-line number, but it's coarse — it doesn't separate "slow to start" from "slow to finish," which have different causes and cures. That separation is what the token-level metrics (TTFT, inter-token latency, time to last token) provide, and why the streaming experience is measured with more than one number. This chapter's job is to insist you always know the decomposition; the next sharpens the generation part of it.

Observed evidence

This project's serving experiments measured latency as a decomposition under load, not a single figure: below the throughput knee, generation time dominated and the queue was near-empty; past the knee, requests piled up and queueing became the dominant segment while per-token generation barely changed. The end-to-end number rose, but the cause moved from generation to queue — visible only because the segments were measured separately. See latency decompose under load →

Mental model

End-to-end latency is the sum of network + queue + prefill/TTFT + decode/generation + post-processing. The model's generation is only one term, and often not the largest. "It's slow" isn't a diagnosis until you know which segment dominates — so always measure the decomposition, not just the total, and fix the segment that actually rules.

Common mistakes

  • Treating end-to-end as "model latency." The model is one segment; network, queue, and post-processing often dominate.
  • Optimising without decomposing. Swapping the model to fix a queue or downstream problem wastes effort on the wrong segment.
  • Ignoring post-processing. Retrieval, execution, and downstream calls after the model are real latency you must count.
  • Reporting only the total. One number hides the segment mix and can't guide a fix.

Practical guidance

  • Instrument each segment — network, queue, prefill/TTFT, generation, post-processing — and attribute the total to them.
  • Diagnose a slow response by finding the dominant segment before choosing a fix; don't reach for the model first.
  • Remember the model's generation is a term, not the whole; queue and post-processing scale with load and system design, not model choice.
  • Carry the decomposition into the SLO — an end-to-end SLO is only actionable if you know which segment threatens it.

Summary

  • End-to-end latency — request-out to response-in — is a sum of segments: network, queue, prefill/TTFT, decode, post-processing.
  • The model's generation is one term, and often not the largest; queue and post-processing frequently dominate.
  • "Slow" isn't a diagnosis until you know which segment rules — measure the decomposition.
  • The next chapter sharpens the generation part into precise token-level metrics.

Knowledge check

A user-facing feature's end-to-end latency is 4 seconds and users complain. Before touching the model, what should you check, and why?

The segment decomposition: how much of the 4 seconds is network, queueing, prefill/TTFT, generation, and post-processing. The model's generation is only one term, and often not the biggest — those 4 seconds could be dominated by a busy queue (fix with capacity or batch tuning), a bloated prompt inflating prefill, a slow downstream call in post-processing, or network for distant users. Swapping the model would waste effort if the time lives elsewhere. Measure which segment dominates, then fix that one.

Why is end-to-end latency, though the number users feel, too coarse to optimise generation with?

Because it collapses "slow to start" and "slow to finish" into one figure, and those have different causes and cures. A big end-to-end number could be a long prefill (slow first token — prompt size, queueing) or a long decode (slow streaming — model size, bandwidth, answer length), which need opposite fixes. End-to-end tells you that it's slow and, decomposed, which segment, but to optimise the generation itself you need the token-level split — TTFT vs inter-token latency vs time to last token — which the next chapter provides.

Related chapters