Part 8 — Production Observability

Production quality signals

Measuring production quality·Evaluation·7 min read

The hardest problem in observability: measuring quality in production, where there's no ground truth. You can't run offline scoring on live traffic, so you infer quality from proxy signals — user behaviour, sampled judgment, reliability rates — each imperfect, but together the only window into whether a live system is actually right.

Operational metrics tell you the system is fast, cheap, and up. They say nothing about whether it's right — and in production you hit a wall the offline half never faced: there's no ground truth. Live requests don't come with correct answers attached, so you can't run exact-match scoring on them. Yet quality is exactly what matters most. The resolution is proxy signals: indirect evidence of quality inferred from behaviour and sampling. None is as clean as an offline score, but together they're the only way to know if a live system is degrading before your customers tell you.

What you will understand by the end

  • Why production has no ground truth, and what that rules out.
  • The main proxy signals: user behaviour, sampled judgment, and reliability rates.
  • Why each is imperfect, and why you triangulate rather than trust one.
  • How production quality signals feed the continuous evaluation loop.

The no-ground-truth wall

Offline, every example had a known correct answer, so scoring was mechanical. Production requests arrive with no label — you don't know the right answer for a live query, so you can't directly score correctness. This is the defining constraint of production quality measurement, and it forces a shift from measuring quality to inferring it from signals that correlate with it.

Key idea

In production there is no ground truth, so you can't score correctness directly — you infer it from proxy signals. Every proxy is imperfect and indirect, so the discipline is to triangulate: watch several signals together, because a real quality problem tends to move several of them, while any single one is noisy.

The proxy signals

Three families, each a different angle on "is this any good?":

  • User behaviour (implicit). What users do with an answer betrays its quality: do they accept it, edit it, retry/regenerate, abandon the session, or succeed at the downstream task? A spike in edits or retries is a strong quality-degradation signal — the subject of user feedback.
  • Sampled judgment. You can't score every request, but you can sample a slice and score it — with a calibrated LLM judge for scale, or human review for the hard cases. This gives an actual quality estimate on a subset, at a cost you control.
  • Reliability-layer rates. The system's own guardrail, retry, and fallback frequencies are quality signals: a rising fallback rate or guardrail-trigger rate means more requests are hitting known-bad paths.
Watch out

Every proxy has a confound. Users edit good answers for style, abandon for unrelated reasons, and don't click thumbs; a sampled judge has its own biases; reliability rates can move for non-quality reasons. So no single signal is proof — a quiet thumbs-down rate doesn't mean quality is fine, and a retry spike might be a UI change. Triangulate across signals and confirm with sampled scoring before you trust a quality story.

Triangulate, and sample to actually measure

Because each proxy is noisy and biased, you don't rely on one — you watch a panel of them and treat a coordinated move (edits up and retries up and sampled score down) as a real signal. And crucially, sampled scoring is your one link back to something like ground truth: by scoring a representative sample (calibrated judge or human), you convert "the proxies look bad" into an actual quality number on a subset. The proxies tell you when to look; the sample tells you how bad.

Observed evidence

This project is offline, so it enjoys the luxury production never has: known ground truth for all 88 tasks, scored by exact match. That's precisely the thing you lose in production — which is why this chapter exists. The project measures quality directly; a live deployment of the same system would have to infer it from edits, retries, sampled judge scores, and the correction-policy trigger rate, because the business questions arriving in production wouldn't come with expected intents attached. Offline quality, measured directly — the thing production must approximate →

Feeding the loop

Production quality signals aren't just a dashboard — they're the detector in the incidents-to-tests loop. A proxy signalling degradation surfaces real failing cases; those get root-caused, captured as test cases, and added to the offline suite — which then guards against that failure forever. Production quality signals are how the offline evaluation set learns what production actually breaks on.

Mental model

Production has no ground truth, so you infer quality from proxy signals — user behaviour (accept/edit/retry/abandon), sampled scoring (calibrated judge or human on a slice), and reliability-layer rates. Each is imperfect and confounded, so triangulate across a panel and use sampled scoring to turn "the proxies look bad" into an actual number. These signals are the detector that feeds real failures back into the offline suite.

Common mistakes

  • Assuming operational green means quality green. Metrics don't measure correctness; you need quality signals.
  • Trusting a single proxy. Each is noisy and confounded; a coordinated move across several is the real signal.
  • No sampled scoring. Without periodically scoring a sample, you can watch proxies but never get an actual quality number.
  • Not feeding signals back. Detected degradation should surface cases that become offline tests, not just a dashboard blip.

Practical guidance

  • Watch a panel of proxies — user edits/retries/abandonment, sampled judge/human scores, reliability-layer rates — and treat a coordinated move as a real quality signal.
  • Sample and score a representative slice regularly (calibrated judge for scale, human for hard cases) to convert proxies into an actual quality estimate.
  • Calibrate any judge you use on production samples, and remember each proxy's confound before acting on it.
  • Wire degradation detection into the incidents-to-tests loop so production failures become offline regression cases.

Summary

  • Production has no ground truth, so quality is inferred from proxy signals, not measured directly.
  • The proxies are user behaviour, sampled judgment, and reliability-layer rates — each imperfect and confounded.
  • Triangulate across a panel and use sampled scoring to turn a signal into an actual quality number.
  • Production quality signals are the detector that feeds real failures back into the offline suite.

Knowledge check

Your operational dashboard is all green, but the edit rate, retry rate, and a sampled LLM-judge score have all moved in the "worse" direction over three days. What do you conclude, and what's the next step?

A coordinated move across multiple proxies is a strong signal of real quality degradation — far more than any one of them alone, since a single proxy (edits, retries) has confounds but three moving together in the same direction is unlikely to be coincidence, and the sampled judge score is a direct (if biased) quality estimate. The green operational metrics just confirm it's a semantic problem, not an outage. Next step: sample the degraded traffic and score it (calibrated judge or human) to quantify how bad it is, then open the failing traces to root-cause it — likely drift, a hosted-model change, or a new input type — and feed the confirmed failures into the offline suite as regression cases.

Why can't you just run your offline exact-match scorer on production traffic to measure quality?

Because production requests arrive with no ground truth — you don't know the correct answer for a live query, and exact-match scoring requires a known reference to compare against. Offline, every example had a canonical expected answer; production queries are novel and unlabelled, so there's nothing to exact-match them to. That's the defining constraint: you must infer quality from proxy signals (behaviour, reliability rates) and from sampled scoring (where a calibrated judge or human supplies the missing judgment on a slice), rather than mechanically scoring every request the way you could offline.

Related chapters