Sample size and uncertainty

Every accuracy you report is computed from a finite sample, so it is an estimate with uncertainty attached — a range, not a point. Ignore that and you'll chase differences that are just noise, declare winners that aren't, and trust small numbers you shouldn't. This closing lesson in evaluation-data design is about reading a score with its error bars on, so your comparisons survive contact with statistics.

What you will understand by the end

• Why a measured accuracy is an estimate with a confidence interval.
• How sample size sets the width of the error bars.
• Why comparing two systems is a paired question, not two separate intervals.
• When a difference between two systems is real versus within-noise.
• How to size a dataset for the decision you need to make.

An accuracy is a range, not a point

Measure a system on 50 examples and get 90%, and the honest statement isn't "the system is 90% accurate" — it's "our best estimate is 90%, and the true value is plausibly somewhere in a band around it." That band is the confidence interval, and it exists because a different 50 examples would have given a slightly different number. The score you report is one draw from a distribution.

Use a Wilson interval, not the textbook ± formula. The familiar p ± 1.96·√(p(1−p)/n) (the Wald interval) misbehaves exactly where evaluation numbers live: at small n, and at accuracies near 100%. On 88 tasks it puts 86/88 at an upper bound of 100.8%, and it collapses a perfect 88/88 to the interval [100%, 100%] — claiming zero uncertainty from 88 examples. The Wilson interval is the same idea done correctly: it stays inside [0, 100], it gives 88/88 an honest [95.8%, 100%], and it is asymmetric near the edges, because a system measured at 95% has more room to be worse than to be better. Every interval in this chapter is a Wilson interval.

Sample size sets the error bars

The width of the interval shrinks as the sample grows — roughly with the square root of the number of examples, so quadrupling the dataset halves the error bars. The practical consequences:

• Small sets have wide bands. On a handful of examples, the interval can be ±15 points or more — wide enough that two very different-looking scores are statistically indistinguishable.
• Diminishing returns. Because of the square-root relationship, going from 25 to 100 examples helps a lot; going from 1,000 to 4,000 helps much less. There's a point where more data barely narrows the band.
• Per-slice bands are wider. A 20% slice of a 100-example set is 20 examples; its error bars are far wider than the overall number's — so per-segment claims need more caution.

   n = 20     ├────────── 90% ──────────┤     wide band: ±~13 pts
   n = 100    ├──── 90% ────┤                 ±~6 pts
   n = 500    ├─ 90% ─┤                        ±~3 pts
              (bands shrink ~ 1/√n — quadruple n to halve the band)

Comparing two systems is a paired question

Here is where most evaluation write-ups — and an earlier version of this chapter — go wrong. Having learned that each score has a band, the tempting next step is to compare the two bands and call overlap a tie. Don't. Those intervals answer a different question than the one you're asking.

A confidence interval tells you where a system's absolute accuracy lies — it accounts for the fact that a different sample of tasks would have given a different score. But when you compare two systems, you did not give them different samples. You gave them the same tasks. The measurements are paired, and the shared difficulty of those tasks is a source of variation that affects both systems identically. Comparing independent intervals throws that away and treats correlated measurements as if they were unrelated, which makes the comparison far too conservative: it will call real differences ties.

The 2×2 that actually answers the question

Put the two systems' per-task outcomes into a table. Only two of the four cells carry information:

	B passes	B fails
A passes	both right — no information	A wins (count b)
A fails	B wins (count c)	both wrong — no information

Tasks both systems get right, and tasks both get wrong, tell you nothing about which is better — they cancel. The entire comparison lives in the discordant cells, b and c. If the two systems were truly equivalent, each discordant task would be a coin flip, so b and c should be roughly equal. McNemar's test asks exactly that: given b + c discordant tasks, how surprising is a split this lopsided under a fair coin? Use the exact (binomial) form — evaluation runs almost always have few discordant tasks, and the chi-square approximation is unreliable when b + c is small.

Report the discordant counts themselves, not just the p-value. "A won 12 tasks that B lost; B won 4 that A lost" is a sentence an engineer can act on — it points straight at 16 specific traces worth reading.

Equal scores are not equal systems

The paired view also exposes something aggregates hide. In this run, claude-haiku and openai-strong both score exactly 84/88 (95.5%). Pairwise: 82 tasks both pass, 2 both fail, 2 haiku-only, 2 openai-only. McNemar gives p = 1.00 — a genuine tie in ranking, correctly called.

But those systems are not interchangeable. They disagree on 4 tasks, failing for different reasons. An aggregate tie is a statement about a score, not about behavior; two systems can arrive at the same number by being good at different things. That difference is invisible in the leaderboard and obvious in the paired table — and it may well matter more to your decision than the score does, if the tasks one fails are the ones you care about.

Beyond pass/fail, and beyond two systems

• Non-binary scores. When the metric is a graded score, a latency, or a cost rather than pass/fail, the paired analogue is a paired bootstrap: resample tasks (not observations), recompute the difference each time, and read the interval off the resulting distribution. Same principle — the pairing stays intact through the resampling.
• Many systems or many slices. Every extra comparison is another chance to find a "significant" difference by luck. Comparing 6 systems is 15 pairs; if all 15 were truly ties, you'd expect 0.75 false positives at the conventional 5% threshold, and — treating the comparisons as independent — roughly a 54% chance of at least one. So a spurious winner is closer to a coin flip than a certainty, which is quite bad enough: it means "we tested everything and found one significant result" is barely evidence at all. Decide your comparisons in advance, or correct for multiplicity, and treat post-hoc slice-hunting as hypothesis generation rather than evidence.
• Significance is not size. A p-value says a difference is probably not zero; it says nothing about whether it's big enough to care about. With a large enough dataset a 0.3-point gap becomes "significant" and remains irrelevant. Always pair the test with the effect size and ask whether that magnitude would change your decision.
• One run per task is its own assumption. These tests treat each task's pass/fail as a fixed observation, when it's actually one draw from a stochastic system. That understates the true uncertainty. Where run-to-run variance could change the conclusion, repeat trials and measure it rather than assuming it away.

Size the dataset for the decision

For a system's absolute accuracy, size to the band you can live with: the interval shrinks like 1/√n, so decide the smallest decision-relevant difference and add examples until the band is narrower than it.

For a comparison, the quantity that matters is not the total dataset size — it's the number of tasks the two systems disagree on. Tasks they both pass contribute nothing. That leads to a sharper rule of thumb, and a sobering one:

If the winner takes…	…the disagreement can't reach p < 0.05 below
every disagreement (100/0)	6 discordant tasks
80% of them	12 discordant tasks
70% of them	25 discordant tasks
60% of them	101 discordant tasks

Read that as a floor, not a power calculation: fewer disagreements than this and the result cannot come out significant no matter how lopsided it looks. Now convert to dataset size by dividing by how often the two systems actually disagree. In this project's run, claude-haiku and openai-strong disagree on 4 of 88 tasks — about 4.5%. To accumulate 25 disagreements at that rate you would need roughly 550 tasks; for a marginal 60/40 edge, closer to 2,200. Strong systems that mostly agree are expensive to separate, and no amount of cleverness in the statistics substitutes for the data.

Mental model

A score is an estimate with a Wilson interval whose width shrinks like 1/√n — that answers "how good is this system?" Comparing two systems is a different question, answered on the tasks where they disagree: count who wins the discordant tasks, test that split with exact McNemar, and size the dataset by how many disagreements you need, not how many tasks you have.

Common mistakes

• Reporting a point number. "90%" hides the band; "90% [84, 94]" tells the truth and stops over-reading.
• Comparing two intervals instead of testing the pair. Overlap is not a test; it silently converts real differences into ties.
• Using the Wald ± formula near 100%. It produces bounds above 100% and calls a perfect score certain. Use Wilson.
• Reporting only a p-value. The discordant counts — "A won 10, B won 0" — are what an engineer can act on.
• Ranking by within-noise gaps. A 1–2 point gap on a small set is often a single task changing hands, not a result.
• Trusting tiny slices, and hunting across many of them. A 20-example segment has wide bands, and inspecting enough segments guarantees a spurious winner.

Practical guidance

• Report each system's accuracy with a Wilson interval — that is its absolute quality.
• Compare two systems with a paired test on the discordant tasks (exact McNemar for pass/fail, paired bootstrap for graded scores) — never by whether their intervals overlap.
• Publish the discordant counts alongside the p-value, and read the traces behind them.
• Check the effect size separately from significance; "real" and "worth acting on" are different findings.
• Pre-declare your comparisons, or correct for multiplicity when you inspect many systems or slices.
• Size for disagreements, not tasks — and repeat trials when run-to-run variance could move the conclusion.

Summary

• A measured accuracy is an estimate; report it as a Wilson interval, which stays sane near 100% where the textbook formula doesn't.
• The band shrinks like 1/√n, with real diminishing returns.
• Systems are measured on the same tasks, so comparison is paired. Overlapping intervals are not a test — they manufacture ties.
• The comparison lives entirely in the discordant tasks: count them, test them with exact McNemar, and report the counts.
• qwen7b vs claude-haiku in this run has overlapping intervals but 10–0 discordant tasks (p ≈ 0.002) — the paired view finds a difference the overlap rule hides.
• Equal scores are not equal systems: claude-haiku and openai-strong both hit 95.5% while disagreeing on 4 tasks.

Knowledge check

On a 60-example held-out set, config A scores 88% and config B scores 85%. A teammate wants to ship A. What's the statistical caution?

First, translate the gap into tasks: 88% and 85% of 60 is roughly 53 and 51 — the entire "3-point lead" is about two tasks. Then ask the paired question the aggregate is hiding: on how many tasks did they actually disagree, and who won those? If A won 2 disagreements and B won 0, that's below the six-disagreement floor — it cannot reach significance, and an exact McNemar test returns p = 0.5. That's a tie.

Note what the wrong reasoning would be: "the confidence intervals overlap, so it's a tie" happens to reach the same verdict here, but by invalid means — it would also have called a decisive result a tie. Get to the answer through the discordant tasks, and read those two traces while you're there; they may be more informative than the score.

Two systems each score 96% on your 200-task suite, and the intervals overlap almost perfectly. Your lead says "identical — pick the cheaper one." Is that right?

The ranking conclusion is right and the equivalence conclusion isn't. Equal accuracy means equal counts of failures, not the same failures. Each system misses about 8 tasks; if those are disjoint sets, the systems behave very differently while scoring identically — exactly what happens in this project's run, where claude-haiku and openai-strong both hit 95.5% yet disagree on 4 tasks.

So build the 2×2. If they fail the same tasks, they really are interchangeable on this dataset and cost is a fine tiebreaker. If they fail different tasks, look at which ones: the cheaper system may be failing the cases you can least afford. Cost is a legitimate tiebreaker only after you've confirmed the tie is behavioral and not merely arithmetic.

You need to detect a 2-point accuracy difference between two strong systems. Why won't 100 examples do, and what's the fix?

Two ways to see it, and the second is the useful one. Superficially, on 100 examples each Wilson band is several points wide, far wider than the difference you're chasing. But the binding constraint is disagreements: a 2-point gap on 100 tasks is two tasks, and even a clean 2–0 sweep of the discordant cells can't clear p < 0.05 — you need at least six disagreements for that, and around 25 if the winner takes only 70% of them.

The fix is to size by disagreement rate, not dataset size. If two strong systems disagree on ~5% of tasks, reaching 25 disagreements takes roughly 500 tasks. If that's out of reach, the honest move is to stop trying to resolve a 2-point gap and decide on something you can measure — latency, cost, failure severity — or on which specific tasks each system fails.

Related chapters

• Development, validation and hidden sets — why each split must be large enough to trust
• Ambiguity and annotation disagreement — the other source of uncertainty in a score
• Aggregate metrics versus individual failures — reading the number and the cases behind it
• Median, p95 and p99 — uncertainty and distributions on the performance side