Part 10 — Integrated Case Studies

Interpreting the 95.5% result

Building, comparing, interpreting·Evaluation·7 min read

Reading the leaderboard honestly. The case study's headline is that two systems tied at 95.5%, the open models improved but stayed below the bar, and a guardrail moved a weak model 25 points. This chapter is how to interpret those numbers with the uncertainty and gate-thinking the book taught.

The six-system comparison produced numbers, and numbers invite over-reading. This chapter interprets the case study's headline result the way the book taught — with uncertainty and gate-thinking in mind. The findings: two systems tied at 95.5%, the open models improved from 3B to 7B but stayed below the bar, and the guardrail added ~25 points to a weak model. Read carelessly these become a false ranking; read honestly they're a routing decision.

What you will understand by the end

  • What "two systems tied at 95.5%" means — and why it's a tie, not a ranking.
  • How to read the open models' improvement against the gate.
  • What the guardrail's +25 says about model vs system quality.
  • How to state the result as a defensible conclusion.

Two systems at 95.5% is a tie

The top of the leaderboard shows two systems at 95.5%, and the honest reading is that they are tied — not that one "edged out" the other. On 88 tasks, the confidence band is several points wide, so systems within a few points of each other are statistically indistinguishable; ranking them by a fraction of a point would be reading sampling noise as signal. The defensible statement is "these two lead; the rest trail," and between the two leaders you decide on other grounds (cost, latency, simplicity), not on a difference smaller than its uncertainty.

Key idea

On 88 tasks, a one-or-two-point gap is inside the error bars, so "two systems tied at 95.5%" is the honest reading — not "System A beat System B." Interpreting the leaderboard means respecting the uncertainty: rank by gaps larger than the noise, and call the rest ties. The precision the numbers display exceeds the precision they have.

The open models improved — but relative to a bar

The open models got substantially better from 3B to 7B, which is a real and expected effect of scale. But the interpretive move is to read that improvement against the gate (release gate thinking): did it clear the bar the leaders set? It didn't — the 7B system improved yet stayed below the top systems. So the conclusion isn't "bigger is better" in the abstract; it's "scaling the open model helped but wasn't enough to reach the leaders on this task." Improvement is measured relative to the decision bar, not in isolation.

The guardrail's +25: model vs system

The correction policy added roughly 25 points to the 3B system, and that number carries a specific interpretation: those 25 points are system quality, not model quality. Reported as one number, a "3B system at ~70%" looks like model capability; separating raw from corrected reveals the model is actually weak (45.5% raw) and a hand-written rule is carrying the system. That distinction is the difference between "we have a good model" and "we have a deterministic patch on a weak model" — completely different engineering situations that the same headline number would hide.

Watch out

The single most common misreading of this leaderboard would be "the 95.5% system is the most accurate model" or "System A beat System B by 1.2 points." Both over-read the numbers: the first credits a system result to a model, and the second treats a within-noise gap as a ranking. Interpret with the two dials (raw vs corrected) and the error bars, or the leaderboard tells you a more precise, more flattering story than the data supports.

Stating it defensibly

Put together, the defensible conclusion is a local routing decision: on this 88-task dataset, two configurations tie at the top (95.5%); a cheap open model plus a targeted guardrail closes much of the gap but stays below them; choose the default on quality-and-cost among the leaders. That's a claim the data supports. What it is not — a universal model ranking — is the next chapter.

Observed evidence

The leaderboard shows all of this at once: two systems at 95.5%, the open models climbing from 3B to 7B but below the top, and the raw-vs-corrected gap that reveals the guardrail's +25 on 3B. Reading it with uncertainty and gate-thinking turns a column of numbers into an honest, bounded conclusion. Read the leaderboard with the error bars in mind →

Mental model

Interpret the 95.5% honestly: two systems at the top are tied (a few points on 88 tasks is within the noise); the open models improved 3B→7B but stayed below the bar (improvement measured against the gate); the guardrail's +25 is system quality on a weak model (read raw vs corrected). The defensible conclusion is a local routing decision, not a model ranking.

Common mistakes

  • Ranking the two leaders. A ~1-point gap on 88 tasks is within the error bars — they're tied.
  • Reading improvement in isolation. The open models got better but didn't clear the bar; measure against the gate.
  • Crediting the +25 to the model. It's system quality from a rule; raw vs corrected shows the model is weak.
  • Over-reading the displayed precision. The numbers show more decimals than the data supports; respect the uncertainty.

Practical guidance

  • Treat top scores within the confidence band as tied; decide between them on cost, latency, or simplicity, not the noise.
  • Read every improvement relative to the decision bar (the gate), not as an abstract gain.
  • Always interpret with the two dials (raw vs corrected) so system quality isn't mistaken for model quality.
  • State conclusions as bounded, local claims the data supports — and hand the "what it doesn't prove" to the next chapter.

Summary

  • Two systems tied at 95.5% — a few points on 88 tasks is within the noise, so it's a tie, not a ranking.
  • The open models improved 3B→7B but stayed below the bar — read improvement against the gate.
  • The guardrail's +25 is system quality on a weak model — visible only via raw vs corrected.
  • The defensible conclusion is a local routing decision, honestly bounded — not a universal model ranking.

Knowledge check

The leaderboard shows two systems at 95.5% and a third at 94.1%. A stakeholder wants to declare the 95.5% systems "the winners" and rank them 1 and 2. What's wrong with each part of that?

Both moves over-read the data. Ranking the two 95.5% systems 1 and 2 treats a within-noise difference as real — on 88 tasks they're identical to the decimal, so there's nothing to rank; they're tied, and you'd choose between them on cost/latency/simplicity, not accuracy. And separating the 94.1% system as clearly beaten is shaky too: 95.5% vs 94.1% is ~1.4 points on 88 tasks, likely inside the combined error bars, so it may also be a tie rather than a clear third place. The honest reading is "a cluster of systems leads at ~94–95.5%, indistinguishable within the noise," and the decision among them rests on secondary criteria — not a precise 1-2-3 ranking the sample size can't support.

The 3B system reaches ~70% with its guardrail. Why is it misleading to call this "a 70% model," and what's the accurate framing?

Because ~25 of those points come from a deterministic correction policy, not the model: the raw 3B model scores 45.5%, and a hand-written rule targeting the ranking-for-breakdown confusion carries the system up to ~70%. Calling it "a 70% model" credits the model for the rule's work and hides that the model is actually weak. The accurate framing separates the two dials: "the 3B system reaches ~70%, of which the model contributes 45.5% and the guardrail ~25 points." That distinction matters because "good model" and "weak model with a deterministic patch" are entirely different engineering situations — one generalises, the other depends on a rule tuned to that model's specific failure mode.

Related chapters