Part 9 — Continuous Evaluation

Prompt and model comparisons

Comparing and evolving·Evaluation·6 min read

Improving a system is a long series of A-vs-B comparisons — this prompt or that, this model or that, guardrail on or off. This chapter is the discipline of doing them honestly: change one thing, hold everything else fixed, run the same evaluation, and only believe a difference bigger than the noise.

Day-to-day, improving an LLM system is not one grand evaluation — it's a stream of small comparisons: is this prompt better than that one, is the 7B model worth it over the 3B, does turning the guardrail on help? Each is an A-vs-B question, and each is easy to get wrong in a way that feels like progress. Doing them honestly — one change at a time, same evaluation, difference bigger than the noise — is the core loop of continuous evaluation, and the skill that separates real improvement from wishful iteration.

What you will understand by the end

  • Why iteration is a sequence of controlled A-vs-B comparisons.
  • The rule that makes a comparison attributable: change one thing, hold the rest fixed.
  • Why the same fixed evaluation must score both sides.
  • When a difference is real versus within the noise.

Change one thing at a time

Recall from The model is only one component that a score belongs to a whole configuration. So if you change two things — a new prompt and a new model — and the score moves, you cannot say which change did it. The move that makes a comparison interpretable is to change exactly one part of the configuration and hold everything else identical:

   A: prompt-v1 · model-X · guardrail-on · temp-0   → 88%
   B: prompt-v2 · model-X · guardrail-on · temp-0   → 91%
        ▲ only the prompt differs → the +3 is attributable to the prompt

Change one knob, re-run, and the delta is caused by that knob. Change several and you get a number you can't attribute to anything — the most common way iteration produces motion without knowledge.

Key idea

A comparison is only interpretable if one thing differs between the two configurations. Every other part — prompt, model, decoding, schema, policy, dataset — must be held fixed. Then the difference in the number is the effect of that one change. This is the same "freeze the configuration" discipline that made the six-system comparison legitimate, applied to every iteration.

Same evaluation on both sides

A comparison is only fair if both sides face the identical evaluation — the same dataset, the same split, the same scoring rule. Score A on one set and B on another and the difference could be the datasets, not the systems. This sounds obvious but slips in subtly: comparing today's config on today's data against last month's number on last month's data is not a comparison, because the data moved underneath it.

Watch out

Comparing against a remembered or historical number is not a controlled comparison — the dataset, scoring, or other config may have drifted since. To compare A and B, run both through the same current evaluation, back to back. "It used to score 90%" is a memory, not a baseline.

Is the difference real?

Every comparison lands on the sample-size question. A measured gap is only meaningful if it's larger than the combined uncertainty of the two estimates. On a small set, a 2-point difference between prompts is almost certainly noise — a different sample would flip it — and acting on it is chasing luck. Before you keep B over A, check that B's advantage exceeds the error bars; if it doesn't, they're a tie, and you decide on other grounds (simplicity, cost) or gather more data.

Observed evidence

This project's entire method is controlled comparison: each experiment changed one knob — prefix caching, max_num_seqs, the guardrail, the model size — on an otherwise-fixed configuration and re-scored on the same 88 tasks, so every delta was attributable to that knob. And the two systems tied at 95.5% are the noise lesson in action: a gap inside the error bars is a tie, not a winner. Controlled changes plus honest error bars is how the results mean anything. See the one-knob-per-experiment comparisons →

Mental model

Iteration is a stream of A-vs-B comparisons. Make each one interpretable by changing exactly one part of the configuration and holding the rest fixed; make it fair by scoring both sides on the same current evaluation; and believe the result only if the gap exceeds the noise. One change, same eval, difference bigger than the error bars.

Common mistakes

  • Changing several things at once. The score moves but you can't attribute it — motion without knowledge.
  • Comparing against a remembered number. The dataset or scoring may have drifted; run both sides through the same current eval.
  • Believing within-noise gaps. A small difference on a small set is sampling luck; check it against the error bars.
  • Different scoring or splits per side. Any difference in the evaluation itself contaminates the comparison.

Practical guidance

  • Change one knob per comparison and hold every other part of the configuration fixed, so the delta is attributable.
  • Run both configs through the same current evaluation back to back; never compare to a historical number.
  • Treat gaps smaller than the combined uncertainty as ties — decide on secondary criteria or gather more data.
  • Record each comparison as (what changed, on what eval, the delta, is-it-significant) so the iteration history is auditable.

Summary

  • Improving a system is a sequence of controlled A-vs-B comparisons.
  • Make each interpretable (change one thing, hold the rest fixed) and fair (same current evaluation on both sides).
  • Believe a difference only if it exceeds the noise; within-error-bar gaps are ties.
  • This project's one-knob-per-experiment method — and its 95.5% tie — is controlled comparison done honestly.

Knowledge check

You switch to a new prompt AND a bigger model at once, and accuracy jumps 6 points. Your teammate credits the new prompt. What's the problem, and how should you have run it?

You changed two things, so the 6-point gain isn't attributable to either — it could be the prompt, the model, or an interaction, and crediting the prompt is a guess. To attribute it, run controlled comparisons that change one knob at a time: same model, old vs new prompt (isolates the prompt); then fixed prompt, old vs new model (isolates the model). Each delta, measured on the same evaluation and checked against the noise, tells you what actually caused the improvement.

A new prompt scores 91% vs the old prompt's 89% on a 60-example set. Ship it?

Not on that alone. On 60 examples the confidence band is several points wide, so a 2-point gap sits inside the combined uncertainty — a different 60 examples could easily reverse it, making this a tie, not a win. Either gather more data to shrink the error bars below the gap, or, if they're genuinely tied, choose on secondary grounds (simplicity, cost, latency). Acting on a within-noise difference is chasing sampling luck, not a real improvement.

Related chapters