Part 10 — Integrated Case Studies

Building the initial dataset

Setting up the case study·Evaluation·6 min read

Every dataset principle from Part 5, applied to one real 88-task set. This chapter walks how the case study's evaluation data was built — the splits, the single-reference ground truth, the private authoring, and the adversarial slice — and why each choice made the resulting numbers trustworthy.

A schema and six systems are worthless without a dataset to measure them on, and the case study's 88-task set is Part 5 made concrete. Every choice in building it — how it's split, where the correct answers came from, how it avoids contamination, and why it has an adversarial slice — was a deliberate application of a dataset principle, and together they're why the experiment's numbers can be trusted rather than merely reported. This chapter walks that construction.

What you will understand by the end

  • The dataset's structure: 88 tasks, split into development, held-out, and adversarial.
  • Why each question has a single canonical expected intent (single-reference ground truth).
  • How private authoring guards against contamination.
  • Why an adversarial slice sits apart from the representative held-out set.

The splits: 88 tasks, three jobs

The dataset is 88 business questions, partitioned by the split discipline:

   88 tasks
   ├── development (20)  — iterate: read failures, shape prompts and the correction policy
   ├── held-out   (60)  — the honest number: touched to report, not to tune
   └── adversarial (8)  — hard boundary cases, scored separately

Prompts and the correction policy were shaped against development; the held-out 60 give the reproducible headline number; the adversarial 8 stress the boundaries without distorting it. Reporting the held-out figure — not the development one the system was tuned on — is what makes the 95.5% defensible.

Single-reference ground truth

Each question was authored with one canonical expected QueryIntent — the single-reference ground truth that the schema design made possible. This is the property that lets the deterministic verifier score by exact match: there's one right answer per question, so scoring is mechanical and reproducible, with zero grader variance. Tasks with many valid answers would have forced subjective scoring; the case study deliberately designed the task to avoid that.

Key idea

The dataset's power comes from single-reference ground truth: each of the 88 questions has one canonical expected intent, authored deliberately. That's what turns scoring into exact match and makes every number in the experiment reproducible to the point. The dataset and the scoring method were designed together.

Private authoring guards against contamination

The questions and their expected intents were hand-authored for this specific schema and domain — not drawn from any public benchmark. That single choice defends against contamination: because the base models could not have seen these exact tasks in their training data, the accuracy numbers measure the task, not leaked test answers. A model can't have memorised data you created and kept private, which is why a hand-built private set is the strongest guard against the invisible inflation of benchmark leakage.

Watch out

Had the dataset been pulled from a public natural-language-to-SQL benchmark, the hosted models' scores could have been inflated by training-data leakage — the questions already in their pre-training — and you'd have no way to see it. The private, hand-authored set is what makes the 45.5%–95.5% spread a measurement of capability on this task rather than a measurement of what leaked. Authoring your own private evaluation data is worth the effort.

The adversarial slice, held apart

The 8 adversarial tasks are deliberately hard boundary cases — the ranking-vs-breakdown distinctions and other near-misses where a small phrasing cue flips the correct pattern. They're kept separate from the held-out 60 for the reason Part 5 gives: adversarial cases are harder than typical traffic, so blending them would understate typical quality and bury the boundary signal. Reported separately, the held-out number estimates ordinary performance while the adversarial slice measures fragility at the boundaries — two numbers, two jobs.

Observed evidence

The split structure is what every headline in this case study rests on: the systems were compared on the same fixed 88-task set with its dev/held-out/adversarial partition, so the numbers are attributable and reproducible. You can browse the trials by split and outcome — the dataset isn't an abstraction, it's the openable substrate of the whole experiment. Browse the dataset by split →

Mental model

The case study's dataset is Part 5 applied: 88 tasks split into development (iterate), held-out (the honest number), and adversarial (boundary stress, scored apart); each question has one canonical expected intent (single-reference ground truth, enabling exact match); and it's privately hand-authored so the base models can't have seen it (guarding against contamination). Every choice makes the numbers trustworthy.

Common mistakes

  • Tuning on the held-out set. Prompts and the policy were shaped on development; reporting the held-out number keeps it honest.
  • Using a public benchmark. Its questions may be in the models' training data, inflating hosted scores invisibly; author privately.
  • Blending adversarial into the headline. It understates typical quality and buries the boundary signal; report it separately.
  • Many-valid-answer tasks. They force subjective scoring; the single-reference design is what enables mechanical exact match.

Practical guidance

  • Split before you tune: iterate on development, report on held-out, keep an adversarial slice separate.
  • Author single-reference ground truth where the task allows — it's what makes deterministic scoring possible.
  • Hand-author privately rather than pulling from public benchmarks, to defend against contamination invisibly inflating scores.
  • Keep an adversarial boundary slice and report it as its own number alongside the representative held-out figure.

Summary

  • The dataset is 88 tasks split into development (20), held-out (60), adversarial (8) — Part 5's discipline in practice.
  • Single-reference ground truth (one canonical intent per question) is what enables exact-match scoring.
  • Private hand-authoring guards against contamination inflating the hosted models' scores.
  • The adversarial slice is held apart and reported separately, so the held-out number stays an honest estimate of typical quality.

Knowledge check

Why does hand-authoring the 88 tasks privately (rather than using a public NL-to-SQL benchmark) matter for trusting the hosted models' scores?

Because public benchmark questions are often already in a model's pre-training data, so a hosted model can score highly partly by having memorised the answers — contamination that inflates the score invisibly and doesn't reflect real task capability. Since the case study's questions and expected intents were authored privately for this specific schema and never published, the base models cannot have seen them, so the 45.5%–95.5% spread measures capability on this task rather than leaked test data. A private, hand-built set is the strongest and simplest guard against benchmark leakage.

The adversarial 8 tasks are scored separately from the held-out 60 rather than merged into one accuracy number. Why?

Because the adversarial tasks are deliberately harder than typical traffic (boundary cases like ranking-vs-breakdown), so merging them would understate typical quality — dragging down the headline number with cases that don't reflect ordinary use — and would bury the boundary signal in the average, hiding exactly where the systems are fragile. Reported separately, the held-out 60 give an honest estimate of performance on representative questions, while the adversarial 8 measure fragility at the boundaries. Two numbers answer two different questions; one blended number answers neither.

Related chapters