Part 4 — Evaluation Data and Dataset Design

Coverage and blind spots

Coverage and uncertainty·Evaluation·7 min read

You can only measure what's in your dataset — so whatever isn't in it is a blind spot, a region of the input space where the system's quality is unknown and failures are silent. This chapter is about mapping the input space, checking coverage against it, and treating production surprises as coverage gaps to close.

Every evaluation has a horizon: it can only tell you about the kinds of input it contains. Everything outside that — a request type you didn't think of, a user segment you didn't sample, a phrasing you never saw — is a blind spot, a region where your measured quality simply doesn't apply and failures happen with no warning light. The dangerous part is that a blind spot looks exactly like success: the dashboard is green because the failing cases were never tested.

What you will understand by the end

  • Why "what's not in the dataset" is unmeasured, and therefore a silent risk.
  • The difference between coverage (what you test) and the input space (what exists).
  • How to map the input space and check coverage against it.
  • Why production failures are usually coverage gaps, and how to close them.

You can only measure what you included

An evaluation number describes performance on the distribution the dataset represents. Push an input from outside that distribution through the system and the number says nothing about it — you have no measurement there at all. A 96% accuracy on a set that omits an entire request type is 96% on the types you included and unknown on the one you left out. The omission doesn't lower the number; it removes that region from the number's scope entirely.

Key idea

Absence of failures in your results is not evidence of absence of failures — it may just mean you never tested where they live. A blind spot produces a confident green number precisely because the failing cases aren't in the set. Coverage, not accuracy, is what tells you how much of reality your number actually speaks for.

Coverage versus the input space

Two things to hold apart:

  • The input space — everything the system could be asked: all request types, segments, languages, lengths, difficulties, edge conditions. This is reality's full range.
  • Coverage — the portion of that space your dataset actually contains.

The gap between them is your blind spots. You can't test infinitely, so some gap is inevitable; the goal is that the gaps fall on low-frequency, low-importance regions, not on something common or critical. A blind spot over a rare, harmless case is fine; one over a frequent request type or a safety-critical input is a latent incident.

   INPUT SPACE (everything that could be asked)
   ┌───────────────────────────────────────────────┐
   │  ███████ covered ███████    ░░ blind spot ░░   │
   │  ██ common cases ██████     ░ untested type ░  │  ← failures here are silent
   │  ███████████████████        ░░░░░░░░░░░░░░░░░   │
   └───────────────────────────────────────────────┘
     accuracy describes this        says nothing about this

Map the space, then check coverage

Finding blind spots is a deliberate act, not something the score reveals:

  • Enumerate the input space along the dimensions that matter — request type, segment, difficulty, length, language, format — and list the values in each.
  • Check the dataset against that map: which cells have examples, which are empty? Empty cells are known blind spots.
  • Prioritise closing them by frequency × importance — a common or critical empty cell is urgent; a rare, harmless one can wait.
  • Watch production for inputs that don't fit any cell you enumerated — those are blind spots you didn't even know to look for.
Watch out

The blind spots that hurt are the ones you didn't know existed — the request type nobody on the team thought to enumerate. Enumerating the input space from the team's imagination inherits the team's assumptions, so it misses the same cases the system does. Real traffic is the corrective: production shows you inputs your map didn't have a cell for.

Production failures are coverage gaps

When a failure appears in production that your evaluation never predicted, the root cause is almost always a coverage gap: that input type wasn't in the set, so its quality was never measured. The fix is not just to patch the bug but to add that region to the dataset, closing the blind spot so it's measured from now on. This is the same loop as turning incidents into test cases, viewed as coverage.

Observed evidence

This project's dataset is built to cover the QueryIntent pattern space — the different query shapes the system must handle — plus an adversarial slice for the boundaries between them. Coverage is defined against the taxonomy: every pattern should have examples, and an untested pattern would be a known blind spot. That's coverage as a deliberate map, not a hope. Browse coverage across the pattern space →

Mental model

You can only measure what's in the dataset; everything else is a blind spot where quality is unknown and failures are silent. Map the input space, check which regions your data covers, and keep blind spots on rare/harmless cases — never on common or critical ones. Production surprises are coverage gaps; close them by adding the region to the set.

Common mistakes

  • Reading a green number as full coverage. It only speaks for the regions you tested; untested regions are unknown, not fine.
  • Enumerating the input space from imagination. It inherits your blind spots; use real traffic to find the cells you'd never list.
  • Leaving blind spots on common/critical cases. A gap over a frequent or high-stakes input is a latent incident.
  • Patching a production failure without adding coverage. The blind spot stays open and the class of failure recurs.

Practical guidance

  • Map the input space along the dimensions that matter and check the dataset cell by cell; empty cells are known blind spots.
  • Prioritise gaps by frequency × importance; tolerate blind spots only on rare, harmless regions.
  • Mine real traffic for input types your map didn't anticipate — the blind spots you didn't know to enumerate.
  • Treat every production surprise as a coverage gap: add the region to the dataset so it's measured going forward.

Summary

  • An evaluation only speaks for the regions its dataset covers; everything else is a blind spot with silent failures.
  • Hold apart the full input space and your coverage; keep the gaps on rare, harmless regions.
  • Map the space and check coverage deliberately — the score won't reveal blind spots, and imagination inherits them.
  • Production failures are coverage gaps; close them by adding the region, not just patching the bug.

Knowledge check

Your eval shows 95% and stakeholders treat the system as broadly reliable. What claim does that 95% actually license, and what doesn't it?

It licenses "95% on the input regions the dataset covers." It does not license "95% across everything users might send" — any request type, segment, or phrasing absent from the dataset is a blind spot where quality is simply unknown, and failures there won't appear in the 95%. To claim broad reliability you'd need to show the dataset covers the important regions of the input space, not just that the covered regions score well.

Why is enumerating your input space from a team brainstorm insufficient for finding blind spots?

Because the brainstorm inherits the team's assumptions — the same assumptions that shaped the system — so it tends to miss exactly the input types the system also fails to handle. The blind spots that cause incidents are the ones nobody thought to list. Real production traffic is the corrective: it surfaces inputs that don't fit any cell the team enumerated, revealing gaps the imagination-based map couldn't.

Related chapters