Queueing and saturation

The throughput knee has a cause, and it's worth understanding on its own, because it explains one of the most counterintuitive facts in operations: a system that's fine at 80% load can fall over at 95%. The culprit is queueing — requests waiting to be served — and its defining feature is that waiting time doesn't grow linearly with load. It creeps, then it runs away. Grasp queueing and you understand why the knee is sharp, why tail latency blows out first, and why "fully utilised" is a trap.

What you will understand by the end

• What queueing is and why it makes latency non-linear in load.
• Why waiting time runs away as offered load approaches capacity.
• What saturation is, and why it's the knee's cause.
• Why you must run with headroom, not at 100% utilisation.

Queueing makes latency non-linear

When requests arrive faster than they can be served, they wait in a queue, and total latency becomes service time + waiting time. The trap is that waiting time is non-linear in load: at low utilisation the queue is nearly always empty and waiting is negligible; as utilisation rises toward capacity, the queue spends more time non-empty, and average wait climbs — then, near capacity, it shoots up. Doubling the load doesn't double the latency; near saturation it can multiply it many times over.

Why it runs away near capacity

The intuition: at high utilisation the server is almost always busy, so a new request almost always finds work ahead of it, and — because arrivals and service times vary — bursts pile up faster than the server can drain them between lulls. The closer offered load gets to 100% of capacity, the less slack exists to recover from a burst, so queues that would have drained now persist and grow. In the limit, as offered load reaches capacity, the queue (and thus latency) tends toward unbounded. That runaway is the knee: throughput can't exceed capacity, so the excess load converts entirely into waiting.

   latency
        │                              ╱ ← runaway near capacity
        │                            ╱
        │                          ╱
        │_____________________ ╱
        │  flat while load ≪ capacity
        └───────────────────────────▶ offered load  (→ capacity)
                             ▲ small increases here cause huge latency jumps

Saturation, and the need for headroom

Saturation is the state past the knee: offered load ≥ capacity, the queue growing without bound, tail latency exploding while throughput sits flat at the ceiling. The practical consequence is that you must run with headroom — a target utilisation comfortably below capacity — so that normal bursts queue briefly and drain, rather than accumulating. How much headroom depends on how bursty your traffic is and how tight your latency SLO is: spikier traffic and tighter tails need more slack. Running "leaner" than that headroom trades a little cost saving for a lot of tail-latency risk.

Mental model

Latency = service + waiting, and waiting is non-linear in load: flat while load is well below capacity, then it runs away as load approaches it. Saturation is the state past the knee — queue unbounded, tail latency exploding, throughput flat. Because the last few percent of capacity is where queueing runs away, you must run with headroom; 100% utilisation has no slack for the bursts that inevitably come.

Common mistakes

• Assuming latency scales linearly with load. It's flat-then-explosive; near capacity, small load increases cause huge latency jumps.
• Targeting 100% utilisation. The last few percent is where queueing runs away; you need slack for bursts.
• Ignoring burstiness. Variable arrivals need more headroom than the average load suggests; the average hides the spikes that saturate.
• Reacting only when the median moves. Queueing hits the tail first; p50 stays fine while p99 is already exploding.

Practical guidance

• Run with headroom below capacity — a target utilisation set by your burstiness and tail-latency SLO, not by squeezing every last percent.
• Watch queue depth (running-vs-waiting requests) and p99 as saturation early-warnings; both move before the median.
• Cap concurrency (a max-batch / admission limit) so bursts queue-briefly-then-shed rather than driving the system into runaway saturation.
• Provision for the peak and the burst, not the average — the average utilisation hides the spikes that actually saturate.

Summary

• Queueing makes latency non-linear in load: negligible while load is low, then it runs away as offered load approaches capacity.
• Saturation (load ≥ capacity) is the knee's cause — queue unbounded, tail latency exploding, throughput flat.
• The last few percent of capacity is where queueing runs away, so you must run with headroom; 100% utilisation has no slack for bursts.
• Watch queue depth and p99 as early warnings; provision for the peak and burst, not the average.

Knowledge check

Your system runs fine at 80% of capacity for months, then during a modest traffic spike to ~95% it suddenly develops multi-second tail latencies. Explain why "just 15% more load" broke it.

Because latency is non-linear in load, and 80%→95% moves you onto the runaway part of the curve. Across most of the utilisation range waiting time is negligible, but near capacity the server is almost always busy and has little slack to drain the bursts created by variable arrivals and service times, so queues that used to clear now persist and grow — and waiting time (hence tail latency) shoots up. The "15% more load" wasn't 15% more latency; it pushed the system close to saturation where a routine burst it previously absorbed now causes a multi-second backlog. The lesson: the last few percent of capacity is a cliff, so run with headroom rather than near 100%.

Why does running a serving system at 100% utilisation for cost efficiency backfire?

Because 100% utilisation leaves zero slack to absorb the normal variation in traffic — arrivals are bursty and service times vary, and a system with no headroom can't drain a burst before the next one arrives, so queues accumulate and tail latency explodes (saturation). The last few percent of capacity is exactly where queueing runs away, so pinning the system there trades a small, steady cost saving for frequent, severe tail-latency violations. Stable tail latency and 100% utilisation are in direct tension; you must run with headroom sized to your burstiness and SLO, accepting some idle capacity as the price of predictable latency.

Related chapters

• Throughput and concurrency — the knee that queueing explains
• Median, p95 and p99 — why the tail blows out first under queueing
• Load testing — measuring where saturation begins
• Capacity planning and SLOs — sizing headroom to your SLO and burstiness