Part 4 — Evaluation Data and Dataset Design

Development, validation and hidden sets

Splits and contamination·Evaluation·6 min read

An evaluation is only honest if the data you tune on and the data you judge on are different. This chapter is the discipline of splits — a development set to iterate, a validation set to choose, and a hidden set you touch exactly once — and the cardinal sin of ever letting the last one leak into the first two.

The fastest way to fool yourself in evaluation is to improve your system against the very examples you later use to grade it. The number goes up, everyone celebrates, and it means nothing — you have measured memorisation, not quality. The defence is boring and non-negotiable: split your data, and keep the set you judge on strictly separate from the set you improve on.

What you will understand by the end

  • The three roles data plays: development, validation, and hidden (held-out).
  • Why each must be a different set of examples, and what each one is for.
  • The one rule that makes an evaluation trustworthy: never tune on the hidden set.
  • What "overfitting to the test set" is and how splits prevent it.

Three sets, three jobs

The same pile of labelled examples serves three completely different purposes, and mixing them corrupts the number:

  • Development set — the examples you look at constantly: iterate prompts, read failures, form hypotheses. You are supposed to overfit to these; they're your workbench.
  • Validation set — the examples you use to choose between options (this prompt vs that, 3B vs 7B, guardrail on vs off). You look at the score, not usually the individual cases, and you pick a winner.
  • Hidden / held-out set — the examples you touch once, at the end, to get an honest estimate of how the chosen system performs on data it has never influenced. This is the number you report.
   DEVELOPMENT            VALIDATION              HIDDEN / HELD-OUT
   iterate & debug        choose between options   final honest number
   look at every case     look at the score        touch once, at the end
   overfit on purpose     risk of tuning to it      never tune on it
Key idea

The value of the hidden set comes entirely from its innocence: it only estimates real-world performance because it never influenced the system. The instant you tune against it — even by picking the config that scores best on it — it becomes a development set, and its number stops being an honest estimate.

The cardinal rule

Every split discipline reduces to one sentence: the set you report on must never have influenced the system you're reporting. Look at development all you want; peek at validation scores to choose; but the hidden set is opened once and never optimised against. Break this and your headline number is inflated by an amount you cannot measure.

Watch out

"Tuning on the test set" rarely looks like cheating. It looks like running the hidden set, seeing config B edged out A, and shipping B. You just used the hidden set to choose — that's tuning, and its number is now optimistic. Use validation to choose; reserve the hidden set to confirm the choice you already made.

What overfitting to a set looks like

Improve a system against a fixed set for long enough and it gets better at those specific examples faster than it gets better in general — you're fitting the quirks of the sample, not the task. On the development set that's fine (it's the workbench). The danger is doing it, knowingly or not, to the set you report — because then the reported number rises while true quality is flat, and you discover the gap only in production.

Observed evidence

This project splits its 88 tasks into development (20), held-out (60), and adversarial (8) for exactly this reason. Prompts and the correction policy were shaped against development; the held-out set gives the honest number; the adversarial set stresses the boundaries separately. Reporting the held-out figure — not the development one the system was tuned on — is what makes the 95.5% defensible rather than flattering. Browse trials by split →

Mental model

Three sets: development (iterate, overfit on purpose), validation (choose between options), hidden (touch once for the honest number). The hidden set's only value is that it never influenced the system — so never tune on it. Choose on validation; confirm on hidden.

Common mistakes

  • One set for everything. Tuning and judging on the same examples measures memorisation, not quality — the number is meaningless.
  • Choosing on the hidden set. Picking the config that scores best on held-out data is tuning on it; use validation to choose.
  • Reporting the development number. That's the set you overfit on purpose; it always flatters. Report held-out.
  • A hidden set too small to trust. An honest number on 12 examples has huge error bars — see sample size.

Practical guidance

  • Split before you start improving anything, and write down which set is which.
  • Iterate on development, select on validation, report on held-out — and open the held-out set as late as possible, ideally once.
  • If you must re-use the hidden set across many experiments, treat repeated peeking as slow contamination and plan to refresh it (next chapter).
  • Keep an adversarial slice separate so hard cases stress the system without distorting the headline number (adversarial cases).

Summary

  • Data plays three roles: development (iterate), validation (choose), hidden (report), and they must be different examples.
  • The hidden set is honest only because it never influenced the system — the cardinal rule is never tune on it.
  • Choose on validation, confirm on hidden; reporting the development number always flatters.
  • This project's dev/held-out/adversarial split is why its headline number is defensible.

Knowledge check

You run the held-out set on five config options and ship the highest-scoring one. What did you just do wrong, and what's the honest number now?

You used the held-out set to choose — that's tuning on it — so its number is now optimistic (you picked the config that happened to score best on that particular sample, partly by luck). The honest thing was to select among the five on the validation set, then run the held-out set once on the single chosen config to confirm. To recover, you'd need a fresh held-out set the chosen config never saw.

Why is it fine — even good — to overfit to the development set?

Because the development set is your workbench: you look at every case, read failures, and shape prompts and policies against them on purpose. Its score was never meant to be an honest estimate of general performance — that's the hidden set's job. Overfitting development is how you learn the system's failure modes; you just never report development's number as if it were unbiased.

Related chapters