Building the initial dataset

A schema and six systems are worthless without a dataset to measure them on, and the case study's 88-task set makes the evaluation-data design concrete. Every choice in building it — how it's split, where the correct answers came from, how it avoids contamination, and why it has an adversarial slice — was a deliberate application of a dataset principle, and together they're why the experiment's numbers can be trusted rather than merely reported. This chapter walks that construction.

What you will understand by the end

• The dataset's structure: 88 tasks, split into development, held-out, and adversarial.
• Why each question has a single canonical expected intent (single-reference ground truth).
• How independent authoring reduces contamination risk.
• Why an adversarial slice sits apart from the held-out synthetic set.

The splits: 88 tasks, three jobs

The dataset is 88 business questions, partitioned by the split discipline:

   88 tasks
   ├── development (20)  — iterate: read failures, shape prompts and the correction policy
   ├── held-out   (60)  — untuned internal check: touched to report, not to tune
   └── adversarial (8)  — hard boundary cases, scored separately

Prompts and the correction policy were shaped against development; the held-out 60 provide the untuned internal generalization result; the adversarial 8 stress the boundaries without blending two different questions. The 95.5% figure is the descriptive overall score across all 88 tasks, while held-out accuracy must be reported separately.

Single-reference ground truth

Each question was authored with one canonical expected QueryIntent — the single-reference ground truth that the schema design made possible. This is the property that lets the deterministic verifier score by exact match: there's one right answer per question, so scoring is mechanical and reproducible, with zero grader variance. Tasks with many valid answers would have forced subjective scoring; the case study deliberately designed the task to avoid that.

Independent authoring reduces contamination risk

The questions and their expected intents were hand-authored for this specific schema and domain — not drawn from an existing public benchmark. That choice reduces one important contamination route: the set was created independently before the evaluation rather than selected from questions likely to appear in the base models' training corpora. It does not prove that no similar wording or concepts were present in training, and publishing the artifact now means future evaluations must treat it as public.

The adversarial slice, held apart

The 8 adversarial tasks are deliberately hard boundary cases — the ranking-vs-breakdown distinctions and other near-misses where a small phrasing cue flips the correct pattern. They're kept separate from the held-out 60 for the reason the adversarial-data lesson gives: adversarial cases are deliberately harder than the held-out slice, so blending them would obscure performance on that slice and bury the boundary signal. Reported separately, the held-out number measures performance on the untuned synthetic slice while the adversarial slice measures fragility at the boundaries — two numbers, two jobs. Neither is presented as an estimate of production traffic.

Mental model

The case study applies the evaluation-data discipline: 88 tasks split into development (iterate), held-out (untuned internal check), and adversarial (boundary stress, scored apart); each question has one canonical expected intent (single-reference ground truth, enabling exact match); and the set was independently hand-authored to reduce the risk that the exact tasks appeared in training. Every choice makes the measurement easier to inspect and interpret.

Common mistakes

• Tuning on the held-out set. Prompts and the policy were shaped on development; reporting the held-out number keeps it honest.
• Using a public benchmark. Its questions may be in the models' training data, inflating hosted scores invisibly; author privately.
• Blending adversarial into the headline. It obscures performance on the held-out slice and buries the boundary signal; report it separately.
• Many-valid-answer tasks. They force subjective scoring; the single-reference design is what enables mechanical exact match.

Practical guidance

• Split before you tune: iterate on development, report on held-out, keep an adversarial slice separate.
• Author single-reference ground truth where the task allows — it's what makes deterministic scoring possible.
• Create the set independently rather than pulling from a known public benchmark, to reduce one contamination risk. Once published, treat it as exposed and use a fresh hidden set for future claims.
• Keep an adversarial boundary slice and report it as its own number alongside the held-out synthetic slice.

Summary

• The dataset is 88 tasks split into development (20), held-out (60), adversarial (8) — the evaluation-data discipline in practice.
• Single-reference ground truth (one canonical intent per question) is what enables exact-match scoring.
• Independent hand-authoring reduces the risk of contamination from reusing a known public benchmark; it does not prove the models saw no similar material.
• The adversarial slice is held apart and reported separately, so performance on the held-out synthetic slice is not conflated with boundary stress.

Knowledge check

Why does creating a synthetic 88-task set (rather than using a public NL-to-SQL benchmark) matter for trusting the hosted models' scores?

Because public benchmark questions are often already in a model's pre-training data, so a hosted model can score highly partly by having memorised the answers — contamination that inflates the score invisibly and doesn't reflect real task capability. Since the case study's questions and expected intents were created independently for this specific schema before the run, the comparison avoids directly reusing a known public benchmark. That reduces one contamination risk; it does not prove that the models never encountered similar concepts or wording. The dataset is public now, so future runs must use a fresh hidden set for an unbiased generalization claim.

The adversarial 8 tasks are scored separately from the held-out 60 rather than merged into one accuracy number. Why?

Because the adversarial tasks are deliberately harder than the held-out synthetic slice (boundary cases like ranking-vs-breakdown), merging them would obscure performance on that slice and bury the boundary signal in the average. Reported separately, the held-out 60 measure performance on untuned synthetic questions, while the adversarial 8 measure fragility at the boundaries. Neither number estimates production traffic. Two numbers answer two different internal questions; one blended number answers neither clearly.

Related chapters

• Development, validation and hidden sets — the split discipline applied here
• Ground truth and reference answers — the single-reference intents
• Holdout contamination — why private authoring matters
• Adversarial and boundary cases — the separate boundary slice