Quality–latency–cost tradeoffs

Almost every serving decision is a trade among three things: quality (how correct), latency (how fast), and cost (how much). You cannot maximise all three at once — pushing one corner pulls the others — so there is no single "best" configuration, only the best point for a given use case. This chapter is the framework for that choice: recognise the triangle, know which way each lever moves it, and pick the operating point on purpose rather than optimising one axis and discovering too late what it cost you on the others.

What you will understand by the end

• Why quality, latency, and cost form a trade-off triangle with no free corner.
• How the main levers move you around it.
• Why the right operating point is use-case-specific, not universal.
• How to choose the point deliberately rather than by accident.

The triangle

                     QUALITY
                    (how correct)
                       /\
                      /  \
                     /    \
                    /      \
              COST /________\ LATENCY
           (how much)      (how fast)

        pull one corner and the other two move — no config wins all three

The three axes are coupled through the levers you've already met:

• Bigger/better model → higher quality, worse latency and cost.
• Smaller/quantized model → lower cost and latency, risk to quality.
• More generation (reasoning, longer outputs, retries) → often higher quality, worse latency and cost.
• Bigger batch / higher utilisation → lower cost, worse latency (past the knee).
• A guardrail / correction → higher quality, small cost/latency add.

Every one of these improves one or two corners at the expense of another. There is no lever that improves all three.

The right point is use-case-specific

Because there's no universal best, the operating point is dictated by what the application actually needs:

• A real-time consumer chat weights latency heavily — a snappy TTFT matters more than the last few points of quality, and cost must stay sane at scale.
• A high-stakes analysis (legal, medical, financial) weights quality above all — slower and pricier is fine if it's more correct, because a wrong answer is catastrophic.
• A high-volume background job weights cost — batch hard, tolerate latency, accept a quality bar that's good enough at volume.

Same three axes, three completely different points. The mistake is importing another use case's operating point — optimising a background job for latency, or a real-time feature for cost — instead of deriving it from your own requirements.

Choose the point deliberately

The framework in practice: state the use case's requirements as constraints and a priority — e.g. "p99 latency < 1s and cost < $X are hard constraints; within those, maximise quality" — then find the configuration that meets the constraints and optimises the priority axis. This turns a vague "optimise" into a concrete search, and it makes the trade explicit so stakeholders agree on what's being given up. The cost-per-success metric often mediates the quality-vs-cost edge; latency SLOs pin the latency corner.

Mental model

Quality, latency, and cost form a triangle with no free corner — every lever (model size, generation length, batch size, guardrails) trades among them. There's no universal best, only the best point for a use case: real-time weights latency, high-stakes weights quality, high-volume weights cost. Choose the point deliberately — constraints plus a priority axis — and always ask what a gain on one corner costs on the other two.

Common mistakes

• Optimising one axis in isolation. Maximising quality or minimising cost silently sacrifices the other corners; check all three.
• Assuming a universal best config. The right point is use-case-specific; there's no configuration that wins the whole triangle.
• Importing another use case's operating point. A background job's cost tuning ruins a real-time feature's latency, and vice versa.
• Trading quality for cost without cost-per-success. A cheaper, worse model can raise cost per useful result — a false saving.

Practical guidance

• State each use case as hard constraints plus a priority axis (e.g. "p99 < 1s and cost < $X; maximise quality within that") and search for the configuration that fits.
• For every lever, evaluate its effect on all three corners, not just the one you're targeting.
• Use cost per success to mediate the quality-vs-cost edge and latency SLOs to pin the latency corner.
• Make the trade explicit so stakeholders agree on what's being given up — different use cases get different, documented operating points.

Summary

• Quality, latency, and cost form a trade-off triangle — no configuration is best on all three.
• Every lever (model size, generation length, batch, guardrails) moves you around it, improving some corners at the expense of another.
• The right operating point is use-case-specific: real-time → latency, high-stakes → quality, high-volume → cost.
• Choose it deliberately — constraints plus a priority axis — and always weigh a gain against its cost on the other two corners.

Knowledge check

A team ships the same "best" configuration — biggest model, long reasoning, max quality — for both a real-time chat feature and a nightly batch-analysis job. Why is this wrong for at least one of them?

Because there's no universal best point on the triangle — the two use cases weight the corners differently. The real-time chat weights latency: the biggest model with long reasoning likely blows the TTFT/end-to-end SLO and costs too much at interactive scale, so it needs a faster, cheaper point even at some quality cost. The nightly batch job weights cost/throughput: it can tolerate latency and should batch hard on a cheaper configuration, so max-quality-max-cost may be wasteful there too. Shipping one config ignores that each use case demands its own deliberate operating point derived from its own constraints and priority axis.

You cut cost by switching to a smaller model. Which two other corners must you check, and what metric guards against a false saving?

You must check quality (the smaller model may be less accurate) and latency (it's usually faster, but confirm it still meets or improves the SLO rather than assuming). The metric that guards against a false saving is cost per successful result (cost per request ÷ accuracy, including the downstream cost of wrong answers): a smaller, cheaper-per-request model that drops accuracy can actually raise cost per useful answer, so the apparent cost cut is illusory. Only if cost per success drops while latency stays within SLO is the smaller model a genuine win on the triangle.

Related chapters

• Cost per successful result — the metric mediating the quality-vs-cost edge
• TTFT, inter-token latency and time to last token — the latency corner's metrics
• Quantization — a classic quality-for-cost/latency lever
• Capacity planning and SLOs — pinning the latency corner as a constraint