Part 10 — Integrated Case Studies

Comparing complete systems

Building, comparing, interpreting·Evaluation·6 min read

The six-system head-to-head at the heart of the case study. This chapter lays out the comparison — which systems, on what shared contract, measured how — and why comparing configurations (not models), some sharing a model and differing in policy, is the only fair way to answer 'which should we run?'

With A1.5 built, the case study runs the comparison it existed to run: six complete systems, head-to-head, on the same task, to answer a real decision — how should we close the semantic-quality gap: a bigger open model, a deterministic guardrail, or a hosted frontier model? This chapter lays out that comparison and why its structure is fair. It's the complete-system principle and the comparison discipline at full scale, on real systems.

What you will understand by the end

  • The six systems compared, and the shared contract they ran on.
  • Why comparing configurations, some sharing a model, is fair — not a rigged fight.
  • How the comparison was measured (same dataset, same deterministic scorer).
  • What kind of decision it's built to answer.

The six systems and the shared contract

The comparison holds the task fixed and varies the system: the same 88-task dataset, the same QueryIntent contract, the same deterministic exact-match scorer. What differs is the configuration:

   Same contract (schema, dataset, scoring) · six systems:
   ├── Qwen 3B  + correction policy     (small open model + guardrail)
   ├── Qwen 7B  + correction policy     (larger open model + guardrail)
   ├── Claude Haiku                     (hosted)
   ├── Claude Sonnet                    (hosted)
   └── two OpenAI models                (hosted)

The systems differ in model and in whether a correction policy is present — deliberately, and legitimately, because the unit of comparison is the system, not the model.

Why the asymmetry is fair, not rigged

It looks uneven — some systems have a guardrail, some don't — and that would be unfair if you were comparing models. But you're comparing configurations, and the question is "which complete system should we run?" A configuration that pairs a cheap open model with a targeted policy is a legitimate contender against a hosted frontier model with none; that's the actual choice a team faces. Comparing systems means each row is whatever combination you'd really deploy, measured as a whole.

Key idea

The comparison is fair precisely because it compares systems, not models. Two rows sharing the same open model with and without a policy, ranked against hosted models with none, is exactly the real decision — "which whole configuration should we run?" — not a rigged model contest. The asymmetry is the point: you compare the things you'd actually deploy.

How it was measured

Every system ran the same evaluation: one attempt per task on the fixed 88-task dataset, scored by the same deterministic verifier by exact match against the expected intent, recording both raw (model alone) and corrected (after any policy) outcomes. Because the dataset, contract, and scorer are identical across systems, every difference in the numbers is attributable to the system, not to the measurement — the same-eval-on-both-sides discipline, applied across six systems at once.

Watch out

The comparison is only fair because the evaluation is identical for every system — same dataset, same contract, same scorer, same one-attempt method. If different systems were scored on different data or by different rules, the ranking would reflect the measurement, not the systems. Holding the evaluation fixed is what makes the six numbers comparable; the systems vary, the yardstick doesn't.

What decision it answers

The comparison is built to answer a routing decision: for this task, which configuration should be the default? It surfaces the real trade — cheap open models plus a guardrail versus pricier hosted models — as measured quality (and, folded in with cost, as cost per success). It does not answer "which model is best in general" — that's the subject of what the experiment does not prove. It answers the concrete, local question a team actually has.

Observed evidence

The leaderboard is this comparison made browsable: six systems, one contract, raw and corrected accuracy per row, every trial openable. It's the whole point of the project — not six models ranked, but six configurations compared on the exact decision of which to deploy for this task. Open the six-system comparison →

Mental model

The case study compares six complete systems — two open models with a policy, four hosted — on one fixed contract (same dataset, schema, deterministic scorer), measuring raw and corrected accuracy each. Comparing configurations (not models), some sharing a model and differing in policy, is fair because the question is "which whole system should we run?" — the real decision — with the evaluation held identical so the systems, not the yardstick, vary.

Common mistakes

  • Reading it as a model ranking. It compares systems (configurations); some rows share a model and differ only in policy.
  • Calling the asymmetry unfair. Comparing whole configurations you'd actually deploy is the point, not a rigged fight.
  • Varying the evaluation across systems. Different data or scoring would make the ranking reflect the measurement, not the systems.
  • Expecting a universal answer. It answers a routing decision for this task, not "which model is best."

Practical guidance

  • Frame your own comparison as systems on one fixed contract — same dataset, schema, and scorer for every contender.
  • Include the configurations you'd actually deploy (model + policy combinations), even asymmetric ones — that's the real decision.
  • Record raw and corrected per system so the policy's contribution is visible.
  • Keep the comparison to a local routing decision ("which config for this task"), not a universal claim.

Summary

  • The case study compares six complete systems — two open models with a policy, four hosted — on one fixed contract.
  • Comparing configurations, not models (some sharing a model, differing in policy) is fair because it mirrors the real "which system to run?" decision.
  • The evaluation is held identical across systems, so differences are attributable to the systems.
  • It answers a routing decision for this task, measured as quality (and, with cost, cost per success).

Knowledge check

Two of the six systems use the same open model, one with a correction policy and one without, ranked against hosted models that have no policy. Why is this a fair comparison rather than a rigged one?

Because the unit of comparison is the system (configuration), not the model, and the question is "which whole system should we run for this task?" A cheap open model paired with a targeted correction policy is a legitimate contender — it's an actual configuration a team could deploy — against a hosted frontier model with none, which is another. Including both, and the with/without-policy variants of the same open model, is exactly the real decision, not a rigged model contest. It would only be unfair if you claimed to be comparing models; comparing deployable configurations, each measured whole on the identical evaluation, is precisely the fair and useful comparison.

What single property of the setup makes the six systems' numbers comparable to each other?

The evaluation is identical for every system: the same 88-task dataset, the same QueryIntent contract, the same deterministic exact-match scorer, and the same one-attempt method. Because the yardstick doesn't change across systems, every difference in the resulting numbers is attributable to the systems rather than to the measurement. If different systems were scored on different data or by different rules, the ranking would partly reflect those differences in the evaluation itself, making the numbers incomparable. Holding the evaluation fixed while varying only the system is what makes the comparison valid.

Related chapters