Building A1.5

The pieces are now in place — a use case, a schema, a dataset, a diagnosis, and a guardrail. This chapter is about how they were assembled into a running experiment: the iteration loop that produced the configuration the project calls A1.5, and the discipline that keeps its numbers honest. It's continuous evaluation and versioning shown as they actually happened, on the way to the comparison.

What you will understand by the end

• What "A1.5" refers to and how it was built.
• The develop-on-dev, confirm-on-held-out loop in practice.
• Why each system is a versioned configuration, not "a model we tried."
• How this sets up the fair six-system comparison.

What A1.5 is

A1.5 is a named version of the complete system — a specific, frozen configuration: a model, the QueryIntent schema, constrained decoding, the prompt, and (for some variants) the correction policy. The "1.5" is a version marker in an iteration history — the point the project reached by taking a baseline system and adding the diagnosed-and-targeted improvements. Naming it as a version, not a model, is deliberate: it's the whole pipeline that was measured, and it's the whole pipeline the number belongs to.

The iteration loop, in practice

A1.5 was reached by the loop the book prescribes, run for real:

• Develop on the development set. Read the failing trials, form the ranking-vs-breakdown hypothesis, and shape the prompt and the correction policy against those 20 dev tasks — the set you're supposed to overfit.
• Change one thing, measure. Each change (one knob at a time) was applied to an otherwise-fixed configuration and re-scored, so every delta was attributable — the guardrail's +25, the model-size effect, and so on.
• Confirm on the held-out set. The resulting configuration was then measured on the untouched held-out 60 to get the honest number — never tuning against it.

Systems, not models

Because A1.5 is a versioned configuration, the experiment could legitimately include variants that differ in more than the model — notably, two systems that use the same open model with and without the correction policy. That's only sensible because the unit being compared is the system: "Qwen 3B + policy" and "Qwen 3B, raw" are different configurations of the same model, and comparing them tells you the policy's effect. Treating each row as a versioned system, not a model name, is what makes the next chapter's asymmetric comparison valid rather than unfair.

Mental model

A1.5 is a versioned configuration (model + schema + decoding + prompt + policy), reached by the develop-on-dev, change-one-thing-and-measure, confirm-on-held-out loop. Naming it a version — not a model — is what keeps its number honest and makes every comparison a comparison of configurations, including variants that share a model but differ in policy.

Common mistakes

• Calling A1.5 "a model." It's a whole configuration; the number belongs to the pipeline, not the model in it.
• Iterating against the held-out set. Tuning on it inflates the reported number; develop on dev, confirm on held-out.
• Changing several things between measurements. You lose attribution; change one knob at a time.
• Comparing configurations by model name. Two systems can share a model and differ by policy — the system is the unit.

Practical guidance

• Treat each experiment run as a named, versioned configuration, and record everything that can move the number.
• Run the develop-on-dev → confirm-on-held-out loop, and report the held-out figure, never the one you tuned on.
• Change one knob per comparison so each delta (guardrail, model size) is attributable.
• Structure the experiment as a comparison of systems, so variants that share a model but differ in policy are fair to include.

Summary

• A1.5 is a versioned configuration (model + schema + decoding + prompt + policy), not a model choice.
• It was built by developing on the dev set and confirming on the held-out set — the honest-number discipline.
• Each change was one knob at a time, so deltas are attributable.
• Treating rows as systems, not models is what makes the upcoming six-system comparison fair.

Knowledge check

Why does the project call its result "A1.5" (a version) rather than naming the model that produced it, and what would go wrong with the model-name framing?

Because the number belongs to the whole configuration — model, schema, constrained decoding, prompt, and correction policy, frozen together — not to the model alone. Naming it a version keeps that explicit and makes comparisons fair: it lets the experiment include variants that share a model but differ in policy (e.g. "3B + policy" vs "3B raw") and compares them as systems. The model-name framing ("Qwen 3B scored X") would credit or blame the model for work the prompt and policy did — hiding, for instance, that ~25 of the 3B system's points came from a hand-written rule, not the model — and would make the asymmetric comparison (some systems have a policy, some don't) look unfair instead of legitimate.

The prompt and correction policy were shaped on the 20 development tasks, but the headline number comes from the 60 held-out tasks. Why this separation?

Because a number is only an honest estimate of performance if it comes from data the system was never tuned on. The development set is the workbench — you read its failures and shape the prompt and policy against it, deliberately overfitting. If you then reported the development number, it would flatter the system. The held-out 60 were untouched during iteration, so measuring the final configuration on them gives a figure that reflects generalisation, not memorisation of the tuning set. Had the project iterated against the held-out set (run, tweak, re-run), that number would be inflated by an unmeasurable amount — the develop-here / report-there separation is exactly what makes A1.5's result defensible.

Related chapters

• Versioning a complete system configuration — why A1.5 is a version, not a model
• Prompt and model comparisons — the change-one-thing loop
• Development, validation and hidden sets — develop-on-dev, confirm-on-held-out
• Comparing complete systems — the six-system comparison this sets up