Part 9 — Continuous Evaluation

Turning incidents into test cases

The compounding loop·Evaluation·6 min read

The single highest-leverage habit in the whole field: every production failure becomes a permanent test case. It closes the loop between reality and your evaluation, grows the dataset along the exact dimensions your system fails on, and guarantees the same failure can never silently happen twice.

If you take one habit from this book, take this one. Every part of the evaluation program so far — datasets, failure analysis, regression suites, gates — becomes dramatically more powerful when fed by a single practice: every production failure is turned into a permanent test case. This is the loop that connects the messy reality of production back to your clean offline evaluation, and it's what makes an evaluation program learn. Systems that do this get monotonically harder to break; systems that don't keep re-shipping the same failures.

What you will understand by the end

  • Why production failures are the best possible source of test cases.
  • The loop: incident → root cause → test case → regression suite → coverage closed.
  • Why this makes the evaluation set grow along your real failure dimensions.
  • How it ties every earlier chapter together into a self-improving program.

Production is your best test-case generator

Every offline dataset is a model of reality, and reality always has blind spots you didn't anticipate. Production is where those blind spots reveal themselves — as incidents: real failures on real inputs you never thought to test. That makes each incident a gift: it's a real, important, and confirmed failing case, exactly the kind of test that's hard to invent because you wouldn't have thought of it. Inventing test cases inherits your assumptions; harvesting them from production escapes them.

Key idea

A production incident is a free, high-value test case — a real input, a confirmed failure, a known-important region of the space. Discarding it after the fire is out means you'll meet the same failure again. Capturing it means that exact failure, and its whole cluster, can never silently recur. Incidents are the one test source that escapes your own blind spots.

The loop

Turning an incident into durable protection is a pipeline that reuses everything in Part 6 and this part:

   INCIDENT             ROOT CAUSE            TEST CASE            SUITE               COVERAGE
   real failure   ─▶    diagnose the    ─▶   the input +    ─▶   added to the   ─▶   blind spot
   in production        true cause           correct answer      regression suite    closed
                        (fail-rootcause)      (+ its cluster)     (cont-regression)   (data-coverage)
        ▲                                                                                  │
        └──────────────── the suite now blocks this failure from ever recurring ──────────┘
  • Incident — a real failure surfaces from production (observability, a user report, a spot-check).
  • Root cause — diagnose it: reproduce, localise, characterise the pattern.
  • Test case — capture the failing input with its correct answer — and, because failures come in clusters, a few cases spanning the boundary, not just the one instance.
  • Regression suite — add them to the suite and the gate, so the failure is now blocked on every future change.
  • Coverage closed — the blind spot that let the incident through is now measured.
Watch out

Fixing the incident without capturing it as a test is the most common way teams re-ship the same failure. The pressure of an incident is to patch and move on — but a patch with no regression test is protected only until the next unrelated change silently undoes it. The incident isn't closed until the test exists. Make "add the regression test" part of the incident's definition of done.

The set grows where it matters

Do this consistently and something valuable happens automatically: your evaluation set grows precisely along the dimensions your system actually fails on. You're not guessing what to test — production tells you, one incident at a time. Over months the suite becomes a dense record of every real-world failure mode, weighted toward exactly the cases that bite, which is a far better test set than any amount of up-front imagination could produce. This is how an evaluation program compounds: each failure makes the next one less likely.

Observed evidence

This project's adversarial slice is this habit in miniature: the boundary cases (like ranking-vs-breakdown) are exactly the failure modes worth pinning as permanent tests, and the hidden test is designed to be refreshed as new failure modes surface. Scale that to "every production failure becomes a case" and the 88-task set becomes a living, growing suite that tracks reality. The captured failures that became permanent tests →

Mental model

Every production incident is a real, confirmed, high-value failing case — capture it. Incident → root cause → test case (plus its cluster) → regression suite → coverage closed, and the failure can never silently recur. The set grows along your true failure dimensions, so the program compounds: each failure makes the next less likely. The incident isn't done until the test exists.

Common mistakes

  • Patch and move on. Fixing an incident without capturing it as a test guarantees it can silently return.
  • Capturing only the single instance. Failures come in clusters; add cases spanning the boundary, not just the one input.
  • Not closing the coverage gap. The blind spot that let the incident through stays open unless you add that region to the dataset.
  • Treating incidents as noise. They're the best test cases you'll ever get — the ones you couldn't have imagined.

Practical guidance

  • Make "add the regression test" part of every incident's definition of done — the incident isn't closed until the test exists.
  • Capture the failing input with its correct answer, plus a few cases spanning the cluster/boundary it represents.
  • Add them to the regression suite and gate, and mark the coverage gap closed.
  • Periodically review the accumulated incident-tests to spot recurring themes worth a deeper fix.

Summary

  • Every production failure becomes a permanent test case — the highest-leverage habit in evaluation.
  • The loop — incident → root cause → test case → regression suite → coverage closed — reuses all of Part 6 and this part.
  • Incidents are the best test source because they're real, confirmed, and escape your blind spots.
  • The set grows along your true failure dimensions, so the program compounds — each failure makes the next less likely.

Knowledge check

A production incident is root-caused and patched, the metric recovers, and the team closes the ticket. What did they skip, and why will it bite them?

They skipped capturing the failure as a permanent test case in the regression suite (and closing the coverage gap). The patch fixes the symptom now, but with no regression test guarding it, the next unrelated prompt tweak or model swap can silently reintroduce the same failure — and no one will notice until it hits production again. The incident isn't truly closed until the failing input, its correct answer, and a few cluster cases are in the suite and gate, so that exact failure can never recur unseen.

Why are production incidents better test cases than ones the team writes from scratch?

Because incidents are real, confirmed, and important, and — crucially — they escape the team's blind spots. Hand-written test cases inherit the same assumptions that shaped the system, so they tend to miss exactly the failure modes the system also misses. Production surfaces inputs nobody thought to test, weighted toward what actually bites users. Harvesting them makes the evaluation set grow along the real failure dimensions rather than the imagined ones — coverage you couldn't have brainstormed.

Related chapters