Part 8 — Production Observability

Token, latency and cost metrics

Tracing LLM systems·Serving·6 min read

Alongside traces you track aggregate operational metrics — the tokens, latencies, and costs of live traffic. This chapter is the LLM-specific metric set, why each one is watched as a distribution not an average, and how these numbers connect production back to the performance and economics you sized offline.

Traces let you investigate one request; metrics let you watch the whole stream. Every LLM system emits a characteristic set of operational numbers — how many tokens it processed, how fast it responded, and how much it cost — and tracking them over live traffic is the monitoring backbone of the service. These are the same quantities you reasoned about on the performance side; this chapter is about capturing them in production, as distributions, so you can see the service's health and its bill at a glance.

What you will understand by the end

  • The core LLM operational metrics: tokens, latency, cost (and derived rates).
  • Why each is watched as a distribution / percentile, not an average.
  • How production metrics connect back to the latency and economics you sized offline.
  • Why these are necessary but not sufficient — they say nothing about quality.

The metric set

  • Tokens. Input and output token counts per request, aggregated over traffic. They drive both cost and latency, so they're the base unit — a creeping rise in prompt tokens (a bloated system prompt, growing context) shows up here first.
  • Latency. The TTFT, inter-token latency, and end-to-end times, plus queueing. This is what users feel, and the primary SLO signal.
  • Cost. Per-request and aggregate spend — for hosted models, token price × tokens; for self-hosted, GPU time amortised over requests. The bill, made visible per request.
  • Derived rates. Throughput (tokens/sec, requests/sec), error rate, and the reliability-layer rates — retry, fallback, and guardrail frequencies — which are early warnings of trouble.
Key idea

Tokens, latency, and cost are the operational vital signs of an LLM system, and they're coupled: output tokens drive both latency and cost, so one metric moving often explains the others. Watching them together — not in isolation — is how you catch a regression (a prompt that doubled in size raises tokens, cost, and TTFT at once).

Watch distributions, not averages

The single most important habit: read these as distributions, gated on percentiles, never as means. Latency especially is skewed — a fine median can hide a terrible p99 — and the users who churn are on the tail. So the dashboard tracks p50, p95, p99 of latency, and you alert on tail movement. The same applies to tokens and cost: a rising p99 output length can blow up tail latency and cost long before the average notices.

Watch out

An average latency that "looks fine" routinely sits on top of a p99 that's violating your SLO for one request in a hundred — and those are the requests users remember and abandon. Gate and alert on percentiles, not the mean. A mean-only dashboard is a dashboard that will surprise you with tail failures it structurally cannot show.

Metrics connect production to what you sized offline

These are the same quantities the performance and economics chapters analysed — now measured on real traffic. The latency decomposition (TTFT = prefill + queueing, ITL = decode) tells you which metric to fix when one drifts; the cost-per-request math tells you what a token-count rise means for the bill. Production metrics are where your offline capacity and cost estimates meet reality — and a gap between them is a signal to revisit the sizing, not to distrust the meter.

Observed evidence

This project's serving experiments did exactly this kind of metric capture: each run scraped the engine's tokens/sec, TTFT, queue depth (running-vs-waiting requests), KV-cache utilisation, and prefix-cache hit rate, and read them together to find the throughput knee. That's production-style observability applied to a benchmark — the metrics told the story (compute-bound knee, near-empty KV pool) that the single accuracy number couldn't. See the serving metrics in action →

They don't measure quality

The essential caveat: tokens, latency, and cost are operational, not semantic. A system can be fast, cheap, and 100% available while its answers are wrong — these metrics would all be green. They're necessary (a system that's slow or broke the bank fails regardless of quality) but far from sufficient. Quality needs the separate signals of the next chapters; never let a healthy metrics dashboard stand in for knowing whether the system is right.

Mental model

Tokens, latency, and cost are the coupled operational vital signs of an LLM service — output tokens drive both latency and cost. Watch them as distributions gated on p95/p99, not averages, because the tail is what users feel. They connect production to the performance and economics you sized offline — but they measure operation, not quality, so green metrics are necessary and never sufficient.

Common mistakes

  • Gating on averages. A fine mean hides a p99 violating your SLO for the requests users remember.
  • Watching metrics in isolation. Tokens, latency, and cost are coupled; read them together to explain a regression.
  • Ignoring reliability-layer rates. Rising retry/fallback/guardrail rates are early warnings a metrics-only view will otherwise miss.
  • Treating green metrics as "healthy." They're operational; a fast, cheap system can still be giving wrong answers.

Practical guidance

  • Track tokens (in/out), latency (TTFT/ITL/e2e + queueing), and cost per request and in aggregate, plus throughput and reliability-layer rates.
  • Gate and alert on percentiles (p50/p95/p99), not means — especially for latency and tail token length.
  • Read the metrics together to attribute a regression (a token rise explaining a cost and latency rise), and back to your offline sizing.
  • Pair the metrics dashboard with quality signals — never let operational health stand in for correctness.

Summary

  • The LLM operational metrics are tokens, latency, and cost (plus throughput and reliability-layer rates), and they're coupled.
  • Watch them as distributions gated on p95/p99, not averages — the tail is what users feel.
  • They connect production to the latency and economics you sized offline.
  • They measure operation, not quality — necessary but never sufficient; pair with quality signals.

Knowledge check

Your cost-per-request and p99 latency both jump 30% the day after a prompt change, while accuracy is unchanged. Using the coupling of the metrics, what's the likely cause?

The prompt change almost certainly increased token counts — a longer system prompt, more retrieved context, or longer outputs. Because output tokens drive both cost and decode latency, a token increase raises cost and p99 latency together, exactly the coupled jump you see, while leaving accuracy untouched (the answers are just as correct, only more expensive and slower to produce). Confirm by checking the token metrics: if input/output tokens rose ~30%, that's the cause, and the fix is trimming the prompt/context or capping output length — not chasing latency and cost as if they were separate problems.

Why isn't a dashboard of tokens, latency, and cost — all green — enough to conclude your LLM system is healthy?

Because those are operational metrics, not semantic ones: they measure how fast, how much, and how cheaply the system runs, not whether its answers are correct. A system can be 100% available, well within latency SLO, and cheap while producing valid-but-wrong outputs — every operational metric stays green because none of them looks at meaning. Operational health is necessary (a slow or broken-bank system fails regardless) but not sufficient; you need separate production quality signals to know whether the system is actually right.

Related chapters