Part 7 — Inference Performance and Economics

Median, p95 and p99

Latency metrics·Core·7 min read

Latency is a skewed distribution, not a single number, so the average systematically lies about what users experience. This chapter is the discipline of percentiles — p50, p95, p99 — why the tail is where users churn, and why every serving SLO is written on the tail, not the mean.

If you report one latency number and it's the average, you are almost certainly misleading yourself. Latency is a skewed distribution — most requests are fast, a few are much slower — and the mean gets dragged around by the slow tail while hiding how bad that tail is. The users who suffer, complain, and churn are on the tail, so the whole industry measures latency in percentiles: p50, p95, p99. This chapter is why, and how to read them.

What you will understand by the end

  • Why latency distributions are skewed, making the average misleading.
  • What p50, p95, p99 mean and what each tells you.
  • Why SLOs and alerts are written on the tail, not the mean.
  • How this connects to the uncertainty discipline you already met.

Latency is skewed, so the average lies

Request latencies aren't symmetric around a mean. Most are quick; a minority are far slower — delayed by queueing, a long prompt, an unlucky batch moment, a retry. That long right tail pulls the average up (so it overstates the typical case) while simultaneously hiding how extreme the tail is (so it understates the worst case). The mean manages to be wrong in both directions at once, which is why it's the wrong summary for latency.

Key idea

For a skewed distribution the average is nobody's experience: it's above what most users feel and below what the unlucky ones feel. Percentiles describe the distribution honestly — the median is the typical case, the tail is the bad case — which is why latency is always reported as p50/p95/p99, never as a mean.

Reading the percentiles

  • p50 (median). Half of requests are faster, half slower — the typical experience. It ignores the tail entirely, so a great p50 says nothing about the worst cases.
  • p95. 19 in 20 requests are at least this fast; 1 in 20 is slower. The start of the tail — the experience a meaningful slice of users regularly hits.
  • p99. 99 in 100 are at least this fast; 1 in 100 is slower. The tail proper — and for a high-traffic service, 1 in 100 is a lot of requests, and often the same users repeatedly.
Watch out

A system with a great p50 and a terrible p99 feels unreliable, because every user eventually hits the tail — and a user who occasionally waits 10× as long remembers that, not the median. "Fast on average" or even "fast at the median" is not a serving guarantee; the tail is where reputation is made and lost. Gate on p95/p99, not p50.

Why the tail, specifically

Two reasons the tail dominates what matters:

  • Users experience the tail repeatedly. In a session of many requests, a user meets the p99 roughly every hundred calls — so over time essentially everyone hits it, and the slow experiences are what they recall.
  • The tail is where systems break. Tail latency balloons first as a system approaches saturation — the p99 blows out long before the median moves — so the tail is also your early warning that capacity is running out.

This is why SLOs are almost always written as tail targets ("p99 < 2s") and alerts fire on tail movement — the mean and even the median would stay green through a tail collapse users are already feeling.

Observed evidence

This project's serving sweep is the tail lesson in motion: below the throughput knee, tail latency stayed controlled; past the knee, aggregate throughput flatlined while the queue grew and tail latency climbed sharply — the average barely moved while p99 blew out. Reading the mean would have said "fine"; reading p99 showed the system had saturated. That gap is exactly why SLOs live on the tail. See p99 blow out past the knee →

The same distribution discipline, twice

You've met this shape before. In sample size and uncertainty, the lesson was that an accuracy is a distribution (an estimate with error bars), not a point. Here it's that a latency is a distribution (with a heavy tail), not a point. Both say: a single number hides the spread, and the spread is where the decisions live — the tail for latency, the confidence interval for accuracy. Reporting distributions, not points, is a habit that spans the whole book.

Mental model

Latency is a skewed distribution, so the average is nobody's experience — above the typical case and below the tail. Read p50 (typical), p95 (start of tail), p99 (the tail proper). Users hit the tail repeatedly and it blows out first under saturation, so SLOs and alerts live on p95/p99, never the mean. A single latency number hides the spread that actually matters.

Common mistakes

  • Reporting average latency. It's dragged by the tail and hides it — misleading in both directions.
  • Gating on p50. A great median can sit on a terrible p99 that every user eventually hits.
  • Ignoring tail growth. The p99 balloons first under saturation; watching only the mean misses the early warning.
  • Treating latency as a point. It's a distribution; the tail is where the SLO and the user experience live.

Practical guidance

  • Report latency as p50/p95/p99 (per metric — TTFT, ITL, end-to-end), never as a mean.
  • Write SLOs on the tail ("p99 < X") and alert on tail movement, since it moves first and is what users feel.
  • Watch the p99 as a saturation early-warning — it blows out before the median as load approaches capacity.
  • Carry the same distribution-not-point discipline you use for accuracy uncertainty over to latency.

Summary

  • Latency is a skewed distribution; the average is nobody's experience and hides the tail.
  • p50 is typical, p95 is the start of the tail, p99 is the tail proper — and 1-in-100 is a lot of requests.
  • The tail is what users hit repeatedly and what blows out first under saturation, so SLOs and alerts live there.
  • It's the same distribution-not-point discipline as accuracy uncertainty, applied to time.

Knowledge check

Your dashboard shows average latency of 400 ms, well under your 1 s target, but support tickets about slowness are rising. How can both be true?

The average hides a heavy tail. Most requests are fast (pulling the mean to 400 ms), but the p95/p99 could be several seconds — and those slow requests are what users notice and complain about. In a skewed distribution the mean sits above the typical case yet far below the tail, so a great average coexists with a terrible p99 that a meaningful fraction of requests (and, over a session, essentially every user) hits repeatedly. Check p95 and p99: they're almost certainly violating the 1 s target even though the average isn't. Gate on the tail, and the tickets and the metric will finally agree.

Why is p99 latency a useful early-warning signal for capacity problems?

Because tail latency balloons first as a system approaches saturation: when load nears capacity, queueing delays hit the unlucky requests long before they affect the median, so the p99 blows out while the p50 (and the average) still look fine. Watching p99 therefore surfaces an impending capacity problem while there's still headroom to act — add capacity, shed load, tune batching — whereas waiting for the median to move means the system is already broadly saturated and most users are affected. The tail is both what users feel and the leading indicator of running out of room.

Related chapters