Part 7 — Inference Performance and Economics

Worked scenario: high-volume classification and embedding backfill

Capacity and a worked scenario·Scenario·13 min read

Design a throughput- and cost-bound offline inference system end to end — two model stages (a classifier and an embedding model) sized and costed separately, then combined — from an ambiguous brief through capacity math, a governing bottleneck, a defensible design, separate quality gates, and a sensitivity analysis, using only durable, vendor-neutral concepts.

This is a worked scenario: a realistic but fictional design problem you reason through from ambiguity to a defensible answer. Every number here is an illustrative assumption, labelled as such and shown with its formula — substitute your own and the method still holds. Nothing here is measured evidence.

1. The brief

A retail search company maintains a large product catalogue. To power category browsing and semantic search, every listing needs two things computed once: a category label from a small classifier, and a vector embedding from an embedding model. Most of the catalogue has never been processed. Leadership wants the cheapest way to compute both for the whole catalogue and keep them current, without hurting tag or search quality.

There is no user waiting on any individual item — this is offline batch work. Note from the start: two model passes per item, with different cost profiles, so they must be sized and costed as two separate fleets.

2. What you're told (SLOs & constraints) (illustrative)

  • Backlog: 800,000,000 listings to process once, through both stages.
  • Deadline: clear the backlog within 20 days.
  • Ongoing: ~5,000,000 new/changed listings per day, indefinitely.
  • No per-item latency SLO — overnight or multi-day completion is fine.
  • Quality floor (hard): classification and search-retrieval quality must each stay at or above the current human-tagging bar (defined per stage in §10).
  • Objective: minimise cost per million items (across both passes) while holding both quality floors.
Key idea

The customer just handed you the corner: cost/throughput is the objective; quality is a floor, not the objective; latency is irrelevant. That single sentence inverts almost every design choice from a latency-sensitive service.

3. What you're not told

Missing information that materially changes the design: the text length per listing; the model and precision that clears each stage's quality bar; the current baseline cost and accuracy; whether listings are multilingual; the hard-tail fraction (ambiguous or low-quality text); available hardware and pricing; and how fresh the daily increment must be (minutes, hours, or next-day).

4. Questions to ask before prescribing

Grouped by what each answer changes:

  • Sizes each fleet: average and p95 tokens per listing? Do both stages run per item, and does each model fit one GPU?
  • Sizes the unit cost: what is the smallest model + precision that clears each stage's quality bar (§10)?
  • Shapes the schedule: may the daily increment complete next-day (batchable) or must it be near-real-time (a different architecture)?
  • Sets the safety net: what fraction is hard-tail, and what is the cost of a wrong tag versus a missed one?
  • Bounds the economics: what is the current $/1M items to beat, and — before assuming a dedicated fleet wins — what does a like-for-like hosted quote actually cost (§9)?

5. Back-of-the-envelope math (two stages)

All figures illustrative; formulas shown so you can re-run them. Each item passes through both stages, so each stage must sustain the full item rate.

Required throughput. Rate = items ÷ window seconds.

backfill rate  r_b = 800,000,000 ÷ (20 × 86,400 s)   ≈ 463 items/s
increment rate r_i = 5,000,000 ÷ 86,400 s            ≈ 58 items/s
sustained during backfill  R = r_b + r_i             ≈ 521 items/s   (per stage)

Per-GPU throughput, per stage (illustrative). The classifier and the embedding model have different costs — the embedding encoder is smaller and cheaper per item:

classifier   T_c = 150 items/s per GPU
embedding    T_e = 600 items/s per GPU

Fleet size — each stage separately, then combined. GPUs = ⌈ R ÷ T ⌉.

classifier fleet  N_c = ⌈ 521 ÷ 150 ⌉ = ⌈ 3.47 ⌉ = 4 GPUs   (util 521÷600 = 87%)
embedding  fleet  N_e = ⌈ 521 ÷ 600 ⌉ = ⌈ 0.87 ⌉ = 1 GPU    (util 521÷600 = 87%)
total backfill fleet  N = N_c + N_e = 5 GPUs

Unit cost — per stage, then summed. At GPU price $2.00/GPU-hour (illustrative; both stages priced the same here — in practice each stage can use different hardware):

classifier: items/GPU-hr = 150 × 3,600 = 540,000 = 0.54 M  →  C_c = $2.00 ÷ 0.54 M = $3.70/1M
embedding : items/GPU-hr = 600 × 3,600 = 2,160,000 = 2.16 M →  C_e = $2.00 ÷ 2.16 M = $0.93/1M
combined marginal cost per item (both passes)  C = C_c + C_e = $4.63 per 1M items
Both passes are counted

The embedding pass is not free because the classifier is "binding." It runs its own fleet (1 GPU here) and adds its own $0.93/1M. Sizing only the binding stage would under-count both GPUs and cost.

Backfill fleet cost + reconciliation. Window = 20 days = 480 hours.

classifier cost = 4 × 480 × $2.00 = $3,840
embedding  cost = 1 × 480 × $2.00 =   $960
total backfill cost = $4,800    over    GPU-hours = (4 + 1) × 480 = 2,400
items in window = 800M backfill + (5M/day × 20) increment = 900M
effective cost over backfill = $4,800 ÷ 900 = $5.33 per 1M items

The marginal cost ($4.63/1M) is the ideal at 100% utilisation; the effective cost ($5.33/1M) is the actual fleet. The gap is provisioning overhead — 87% utilisation plus rounding each stage's GPU count up. (Ideal GPU-hours, no rounding: 900M/0.54M + 900M/2.16M = 1,666.7 + 416.7 = 2,083 GPU-hours; the extra 317 hours are the overhead.)

Headroom, stated precisely. Each stage's window capacity ≈ 4×150×1,728,000 = 1.037 B (classifier) and 1×600×1,728,000 = 1.037 B (embedding), both ≥ 900 M needed. So unused capacity ≈ 13% of provisioned capacity (137M ÷ 1,037M), equivalently provisioned capacity is ≈ 15% above demand (1,037M ÷ 900M = 1.15).

6. The governing bottleneck

Cost/throughput-first, quality-floored. The binding hardware resource is compute (both stages are prefill-dominated — a short classify and a single embedding forward pass, with negligible generation), and memory capacity sets the batch you can hold. Master levers, by impact: model size per stage (smallest that clears each eval), quantization (higher low-precision tensor throughput), batching to saturation (no latency SLO to protect), replicate-not-shard (both models fit one GPU), and cheapest $/item hardware per stage (decided per item, not per hour). See the roofline chapter.

7. Design options

  • A — one large general model for both tasks. Simple to operate; one deployment.
  • B — two right-sized models (a small classifier + a dedicated embedding model), each FP8 and eval-gated, batched to saturation, replicated across cheap GPUs, fed by an async queue; a time-boxed burst fleet for the backlog and a scale-to-zero / scheduled job for the daily increment.
  • C — a hosted per-call inference API for both tasks; no fleet to run.

8. Recommended design — B, with justification

Right-size each stage, then push cost:

  1. Smallest classifier and smallest embedding model that each clear their own floor — the single biggest cost lever; capability, not size, is the target.
  2. FP8 precision per stage, eval-gated. On this compute-bound, prefill-dominated work the speedup comes primarily from the GPU's higher low-precision tensor throughput (more operations per second at FP8), not merely from moving fewer bytes. The assumed ≈2× is an illustrative figure gated on the eval, not a universal rule — measure it. See the quantization chapter.
  3. Batch to the throughput roofline — with no latency SLO, operate at saturation, not left of the knee.
  4. Replicate, don't shard — each model fits one GPU, so scale-out is linear.
  5. Async queue + workers — decouple ingestion from processing; autoscale on queue depth.
  6. Two schedules: a 5-GPU time-boxed burst (4 classifier + 1 embedding) clears the backlog in the window; a small steady state (1 + 1 GPU, or scheduled / scale-to-zero) keeps up with the 58 items/s increment, with the one-time backfill tuned independently of the daily-freshness path.

Justification: it hits the objective (lowest combined $/1M) by attacking the two things that move it — per-stage model size and low-precision throughput — while the async/burst structure matches spend to the deadline instead of running a peak fleet forever.

9. Rejected alternatives — and why

  • A (one large model): pays big-model cost on every one of ~900 M items for two narrow tasks that small models clear. Wrong objective — overkill inflates $/1M.
  • C (hosted per-call API): don't dismiss it by reflex. Keep it as a baseline and get a real, like-for-like quote, then compare total cost: token/batch pricing versus a dedicated fleet's achievable utilisation, plus operational overhead, idle time, and data egress. At high, steady, batchable volume a saturated dedicated fleet is often cheaper — but that must be shown, not assumed.
  • Sharding the model across GPUs: unnecessary — both models fit one GPU, so replicate.
  • Speculative decoding: inapplicable here — these encoder-style classification and embedding tasks perform essentially no autoregressive decode, so there is no token-by-token generation for it to accelerate. See the speculative-decoding chapter.
  • Assuming a premium GPU is cheaper: decide on measured $/item per stage, not $/hour — a pricier card only wins if its throughput gain outruns its price.

10. Quality floor & evaluation plan (separate gates per stage)

The two stages are measured differently and gated independently:

  • Classifier — per-class metrics. A golden set with human labels; per-class precision/recall (and confusion analysis), watching rare classes that an aggregate accuracy would hide. FP8 and model size are accepted only if every class stays at or above the floor.
  • Embeddings — retrieval metrics. A labelled query set scored with recall@k / nDCG; the floor is retrieval relevance, a different measurement from classification accuracy.
  • Hard-tail handling: low-confidence items route to review or a stronger model — but calibrate confidence first. Raw classifier softmax scores are often miscalibrated, so check reliability on the golden set (e.g. temperature scaling) before wiring a confidence threshold to escalation; otherwise you will over- or under-escalate.
  • Ongoing: sample-QA shipped items and monitor each stage for drift as new product types appear.
Key idea

Cost-first is not quality-blind. You push cost as hard as each stage's eval allows and stop the moment any class (classifier) or query slice (embeddings) drops below its floor. The evals are what let you quantize and shrink each model safely.

11. Metrics & observability

Items/hour and combined $/1M items (the objective) · per-class precision/recall and retrieval recall@k/nDCG (the two floors) · per-stage GPU utilisation · queue depth and age (is the increment keeping up?) · % routed to review and its calibration · drift signals per stage.

12. Failure modes & operational risks

  • Silent quantization loss in one class or one query slice — caught only by per-class / per-slice evals, not aggregate scores.
  • Miscalibrated confidence — escalation fires on the wrong items until confidence is calibrated.
  • Increment falls behind — queue age grows; a steady stage is undersized.
  • Over-provisioning — integer fleet rounding and low utilisation quietly inflate the bill; watch utilisation, not just throughput.
  • Interrupted burst capacity — acceptable because batch work is interruption-tolerant (checkpoint the queue, retry a reclaimed item); unacceptable on a latency-critical path.
  • Embedding staleness — a model or preprocessing change forces re-embedding; plan a re-index path (this is an embedding concern, independent of the classifier).

13. Sensitivity analysis — what flips the decision

Assumption changes Effect on the design
Tokens/item 400 → 800 both per-GPU throughputs halve (T_c 150→75, T_e 600→300) → N_c ⌈521/75⌉=7, N_e ⌈521/300⌉=2 → 9 GPUs, combined unit cost ~2×
Quality floor rejects FP8 for a stage that stage's throughput ~halves → more GPUs and ~2× its unit-cost contribution (the other stage unaffected)
Deadline 20 → 10 days r_b doubles to ~926/s, R≈984 → N_c=7, N_e=2 → 9-GPU peak; GPU-hours 2,400 → 2,160 (see §14)
A larger model clears accuracy higher $/item for that stage — re-check whether the quality gain is worth it, per class
A latency SLO appears on the increment the batch-to-saturation design breaks; the online path moves left of the knee — a different design
Illustrative, not measured

Every number here is an assumption chosen to make the arithmetic clear. In a real engagement you would measure each stage's per-GPU throughput and each stage's quality on the actual models before committing to a fleet or a precision.

14. Decision exercise

(a) Leadership moves the backfill deadline from 20 days to 10 days. Recompute the required rate, each stage's peak fleet at T_c = 150 and T_e = 600, the total peak fleet, the total GPU-hours, and the effective $/1M at $2.00/GPU-hour. Does the marginal $/1M change? Why does the effective $/1M change even though the work is fixed?

(b) Halfway through, the per-class classification eval shows one product class (say, a niche category) has dropped below its precision floor at FP8, while every other class and the embedding retrieval metric pass. What is the cheapest correct fix, roughly what does it add, and which stage does it touch?

15. Model answer

Show the worked answer

(a) r_b = 800M ÷ (10 × 86,400) ≈ 926/s; with increment R ≈ 984/s. Peak fleets: N_c = ⌈984/150⌉ = 7, N_e = ⌈984/600⌉ = 2 → 9 GPUs peak. GPU-hours = (7 + 2) × (10 × 24) = 9 × 240 = 2,160; cost = 2,160 × $2 = $4,320; effective = $4,320 ÷ 900 = $4.80/1M.

The marginal $/1M is unchanged ($4.63) — it depends only on per-GPU throughput and price, not the deadline. The effective $/1M drops ($5.33 → $4.80) even though the work is fixed, because the ideal (un-rounded) GPU-hours are fixed at ~2,083, and integer fleet rounding is what varies: the 20-day plan rounds 3.47→4 and 0.87→1 (2,400 hours, ~15% overhead); the 10-day plan rounds 6.56→7 and 1.64→2 (2,160 hours, ~4% overhead). Shorter deadline ⇒ larger fleets ⇒ proportionally less rounding waste ⇒ lower effective cost — at the price of a much bigger peak fleet (9 vs 5 GPUs). So "GPU-hours barely change" would be wrong: they fall ~10% purely from rounding.

(b) Do not revert the whole classifier fleet to FP16 — that doubles the classifier cost for every class to fix one, and it touches the wrong lever for a single-class miss. Segment on the failing stage: route just that class through a higher-precision (or larger) classifier path, or apply a targeted relabel/threshold for it, and re-gate on its per-class metric. Added cost ≈ that class's share of volume × the extra classifier unit cost — the classifier stage only. The embedding stage is untouched (its retrieval metric passed; "re-embedding" would be the wrong response to a classification failure). Lesson: hold the floor by segmenting the exception on the stage that failed, never by globalising the fix or conflating the two stages.

16. Related Study chapters