Part 7 — Inference Performance and Economics

Cost per request

The economics·Serving·7 min read

Turn performance into money. Cost per request is where tokens, throughput, and the hosted-vs-self-hosted choice become a line on a bill. This chapter is how to compute it for both models of serving, and why output length and utilisation are the two levers that dominate it.

Performance metrics matter because they turn into money, and the unit that connects them is cost per request. It's what decides whether a feature is economically viable, which system is cheaper to run, and where to spend optimisation effort. The calculation differs between hosted and self-hosted serving, but both come down to the same two levers — how many tokens and how well you use the hardware — and getting the arithmetic right is what keeps a launch from being a surprise on the invoice.

What you will understand by the end

  • How to compute cost per request for hosted and self-hosted serving.
  • Why output tokens dominate the hosted bill, and utilisation dominates the self-hosted one.
  • Why cost per request is coupled to the latency and throughput you already measured.
  • Why it's necessary but not the final economic metric.

Two ways to compute it

  • Hosted (per-token pricing). Cost per request = input_tokens × input_price + output_tokens × output_price. Output tokens are usually priced several times higher than input, so the output length dominates. Straightforward to compute and directly attributable per request.
  • Self-hosted (amortised GPU cost). You pay for GPU-time, not tokens, so cost per request = GPU hourly cost ÷ requests served per hour. That denominator is your throughput, which depends entirely on how well you use the hardware — a GPU serving near its knee has a low cost per request; an idle one has a ruinous cost per request for the same hardware bill.
Key idea

Hosted cost is tokens × price — you pay per unit of work, so cost scales with output length. Self-hosted cost is fixed GPU spend ÷ throughput — you pay for the hardware regardless, so cost per request falls as utilisation rises. Same request, two very different cost structures — and two different levers (trim tokens vs raise utilisation).

The two dominant levers

  • Output length (hosted especially). Because output tokens are the expensive term, a prompt that elicits a shorter, equally-correct answer cuts cost directly — the same lever that cut latency. Verbose outputs are a cost leak; capping max tokens and prompting for concision is often the cheapest optimisation there is.
  • Utilisation (self-hosted especially). Since the GPU bill is fixed, cost per request is inversely proportional to throughput. Running near the knee spreads the fixed cost across many requests; running a GPU at low utilisation means each request carries a huge share of an idle machine. This is why the break-even for self-hosting is a volume.
Watch out

The headline token price or GPU hourly rate is not your cost per request. For hosted, a low per-token price on verbose outputs can still be expensive — output length, not the sticker price, drives the bill. For self-hosted, a "cheap" GPU run at low utilisation can cost more per request than a pricier one run near its knee. Compute the actual per-request number from tokens and throughput, not from the rate card.

It's coupled to performance

Cost per request isn't a separate concern from the latency and throughput you measured — it's the same quantities in money. Output length drives both TTLT and hosted cost; throughput/utilisation drives both the knee and self-hosted cost. So a performance optimisation is usually a cost optimisation and vice versa: shortening outputs helps latency and cost together; running near the knee maximises throughput and minimises per-request cost together. The metrics you already have are most of the economics.

Observed evidence

This project spans both cost structures on one task: the hosted frontier models bill per token, while the self-hosted open models (Qwen on an L4) bill by GPU-time amortised over throughput — which is exactly why the serving sweep's knee matters economically, since cost per request on the L4 is the fixed GPU cost divided by the throughput at the operating point. The compact QueryIntent output also keeps the hosted token bill low. Performance and cost, measured as one. Hosted per-token vs self-hosted per-GPU-hour, on one task →

Not the final number

Cost per request is necessary but incomplete, because it counts every request equally — including the ones that produced a wrong answer. A system that's cheap per request but often wrong may be expensive per useful result. That gap — cost per request vs cost per successful result — is the subject of the next chapter, and it's where economics and evaluation finally meet.

Mental model

Cost per request is performance in money: hosted = tokens × price (output length dominates); self-hosted = fixed GPU cost ÷ throughput (utilisation dominates). The two levers are trimming output tokens and raising utilisation — the same levers that improve latency and throughput. Compute the real per-request number from tokens and throughput, not the rate card — and remember it counts wrong answers too.

Common mistakes

  • Reading cost off the rate card. Output length (hosted) and utilisation (self-hosted) drive the real per-request cost, not the sticker price.
  • Ignoring output length. Verbose answers are a direct cost leak; concision is a cheap optimisation.
  • Running self-hosted GPUs at low utilisation. Fixed cost ÷ low throughput = ruinous cost per request; the break-even is a volume.
  • Stopping at cost per request. It counts wrong answers as if they were useful; the real metric is cost per success.

Practical guidance

  • Compute cost per request the right way: tokens × price (hosted) or GPU cost ÷ throughput (self-hosted) — from measured numbers.
  • Attack the dominant lever: trim output length (hosted) and run near the knee (self-hosted) — both also help performance.
  • Cap max output tokens and prompt for concision as a standing cost (and latency) control.
  • Carry cost per request forward to cost per successful result — the number that accounts for quality.

Summary

  • Cost per request connects performance to money: hosted = tokens × price (output length dominates); self-hosted = fixed GPU cost ÷ throughput (utilisation dominates).
  • The two levers — trim output tokens and raise utilisation — are the same ones that improve latency and throughput.
  • Compute it from measured tokens and throughput, not the rate card.
  • It's necessary but incomplete — it counts wrong answers equally, which the next chapter fixes.

Knowledge check

Two teams serve the same model. Team A uses a hosted API; Team B self-hosts on a rented GPU. Each wants to lower cost per request. What's the primary lever for each, and why do they differ?

Team A's primary lever is output length (and prompt size): hosted cost is tokens × price with output tokens priced highest, so trimming verbose answers and capping max tokens directly cuts the bill. Team B's primary lever is utilisation: self-hosted cost is the fixed GPU spend ÷ throughput, so the bill is paid regardless of tokens and cost per request falls as they push throughput toward the knee — an idle GPU makes every request expensive. They differ because hosted bills per unit of work (scale with tokens) while self-hosted bills for the hardware (scale inversely with how well it's used).

A hosted model advertises a very low per-token price, but your monthly bill is high. Accuracy is fine. What's the likely cause and fix?

Output length. The per-token sticker price is low, but if your system generates verbose outputs (long explanations, unnecessary prose), the output-token term — priced highest — dominates the bill regardless of the low unit price. Cost per request is tokens × price, so many tokens at a low price still adds up. The fix is to trim output length: prompt for concise answers, return compact structured output instead of prose, and cap max output tokens. This cuts cost directly (and improves TTLT), with no model change and no accuracy loss.

Related chapters