Connecting Evaltude to production observability

This is where the book ends, and fittingly it ends by connecting its two halves. The case study is an offline experiment — fixed data, known ground truth, reproducible numbers. A real deployment of the same system lives in production, where there is no ground truth and quality must be inferred from signals. The final move is to connect them: the offline evaluation and the live system are not two separate worlds but one continuous loop, and understanding that loop is understanding the whole discipline this book has been about.

What you will understand by the end

• Why the offline case study is a snapshot, and what production adds.
• How observability replaces the ground truth production doesn't have.
• The loop: offline gate → production → signals → back to offline.
• Why this is where the entire book comes together.

The offline experiment is a snapshot

Everything the case study did — the comparison, the 95.5%, the provisional decision — is a snapshot taken in the lab: fixed systems, a fixed 88-task dataset, known correct answers, run before anything served a real user. That's its strength (reproducible, safe, attributable) and its limit (it measures a model of reality, not reality). Deploying the chosen configuration means stepping out of the snapshot into a live system that meets inputs the dataset never contained and drifts over time.

Production replaces ground truth with observability

The moment the system serves real traffic, the case study's greatest luxury — a known correct answer for every question — disappears. Live business questions arrive unlabelled, so you can't exact-match them. This is exactly the gap Part 8 fills: you instrument the live system (traces, prompt/response capture), infer quality from proxy signals (user edits, retries, sampled judgment, the correction policy's trigger rate), and watch for drift. Observability is how you keep measuring a system whose ground truth you no longer have.

The loop that connects them

Offline and production form a single cycle, and every part of this book is a station on it:

   OFFLINE  ──build & evaluate (Parts 3–7,10)──▶  GATE  ──ship the chosen config──▶  PRODUCTION
      ▲                                                                                  │
      │                                                                    observe (Part 8):
      │                                                                    traces, quality signals, drift
      │                                                                                  │
      └──────── new failing cases become offline tests (Part 9) ◀── incidents ───────────┘

• Offline builds, diagnoses, and gates the system (Parts 3–7, this case study).
• The gate ships the provisional-best configuration.
• Production serves real traffic; observability detects what the lab couldn't — novel inputs, drift, quality slides.
• Those incidents become new offline test cases, and the next hidden test is designed around them.
• The loop repeats — each turn, the offline evaluation is stronger and the live system better understood.

Where the whole book comes together

Trace the loop and you've traced the book. Applications and inference build and run the system; complete systems assemble it; the evaluation program (Parts 4–6) gates it offline; continuous evaluation keeps it good; observability watches it live; performance and economics price it; and training explains where its weights came from. The case study is where all of it touches one real system — and this final connection, offline to production, is the loop that all of it serves. Evaluation isn't a phase; it's the discipline that runs the whole cycle.

Mental model

The offline case study and a production deployment are one system in two settings, joined in a loop: offline builds and gates it (with ground truth); production runs it (with observability replacing ground truth); incidents become new offline tests; repeat. Observability is how you keep measuring without labels; the loop, not the leaderboard, is the deliverable — and it's where the whole book comes together.

Common mistakes

• Treating the offline result as the finish line. It gates a decision; it doesn't certify the system in production forever.
• Shipping and stopping measurement. Production has novel inputs and drift the snapshot never saw; observability must continue.
• Expecting exact-match scoring in production. There's no ground truth for live traffic; infer quality from signals and sampling.
• Breaking the loop. Incidents that don't become offline tests recur; the offline set only improves if production feeds it.

Practical guidance

• Treat the offline gate and production observability as one loop — ship on the gate, then keep measuring live.
• In production, instrument traces and infer quality from signals (edits, retries, sampled judgment, policy-trigger rate), since ground truth is gone.
• Feed production incidents back into the offline suite and design the next hidden test around them.
• Read evaluation as the discipline that runs the whole cycle, not a one-time phase before launch.

Summary

• The offline case study is a snapshot; production is where the system actually lives — with reality but no ground truth.
• Observability (Part 8) replaces the ground truth production lacks: instrument, infer quality from signals, watch drift.
• Offline and production form one loop: gate offline → ship → observe → incidents become offline tests → repeat.
• This is where the whole book comes together — every part is a station on that loop, and evaluation is the discipline that runs it.

Knowledge check

You deploy the case study's 95.5% configuration to production. What capability that the offline experiment relied on do you immediately lose, and how does the book say to cope?

You lose known ground truth — every offline task had a canonical expected intent, so you could exact-match and get a reproducible accuracy; live business questions arrive unlabelled, so there's no correct answer to score against directly. The book's answer is production observability (Part 8): instrument the live system with traces and prompt/response capture, and infer quality from proxy signals (user edits, retries, abandonment, sampled LLM-judge or human scores, the correction policy's trigger rate) plus drift detection, triangulating across them. You then feed the failures those signals surface back into the offline suite as new test cases, so the offline evaluation keeps improving even though you can no longer score production directly.

Why is "the loop, not the leaderboard" the right way to summarise what the case study — and the book — deliver?

Because the leaderboard (the 95.5%) is a snapshot: a reproducible, bounded, provisional decision at one moment on one dataset. The real deliverable is the continuous loop it sits in — build and diagnose offline, gate and ship, observe in production where ground truth is gone, turn incidents into new offline tests, refresh the hidden set, repeat — because that's what keeps a live system good as inputs drift and the frontier moves. A leaderboard certifies a choice once; the loop is the ongoing discipline that makes and re-makes that choice trustworthily. Every part of the book is a station on that loop, which is why the loop, not any single number, is what the whole thing was building toward.

Related chapters

• Monitoring versus observability — the production half of the loop
• Production quality signals — inferring quality without ground truth
• Turning incidents into test cases — feeding production back to offline
• The model is only one component — where the case study's system view began