Prompt and model comparisons

Day-to-day, improving an LLM system is not one grand evaluation — it's a stream of small comparisons: is this prompt better than that one, is the 7B model worth it over the 3B, does turning the guardrail on help? Each is an A-vs-B question, and each is easy to get wrong in a way that feels like progress. Doing them honestly — one change at a time, same evaluation, and a hard look at which tasks actually changed hands — is the core loop of continuous evaluation, and the skill that separates real improvement from wishful iteration.

What you will understand by the end

• Why iteration is a sequence of controlled A-vs-B comparisons.
• The rule that makes a comparison attributable: change one thing, hold the rest fixed.
• Why the same fixed evaluation must score both sides.
• Why a controlled comparison is paired, and why the delta is the wrong place to look.
• Why a zero delta does not mean "no effect."

Change one thing at a time

Recall from The model is only one component that a score belongs to a whole configuration. So if you change two things — a new prompt and a new model — and the score moves, you cannot say which change did it. The move that makes a comparison interpretable is to change exactly one part of the configuration and hold everything else identical:

   A: prompt-v1 · model-X · guardrail-on · temp-0   → 88/100
   B: prompt-v2 · model-X · guardrail-on · temp-0   → 91/100
        ▲ only the prompt differs
          → whatever changed hands is attributable to the prompt
            (subject to run-to-run variability — see below)
          → fixed 5, broke 2   (the "+3" is just these two, subtracted)

Change one knob, re-run, and the outcomes that moved are attributable to that knob. Change several and you get a number you can't attribute to anything — the most common way iteration produces motion without knowledge.

One caveat travels with that word attributable, and it matters for everything below. Holding the rest of the configuration fixed rules out other configuration changes as the cause; it does not rule out chance. A hosted model asked the same question twice can answer differently, so with a single attempt per task some of the outcomes that changed hands would have changed anyway. When that could affect your conclusion, repeat both configurations and compare paired outcome rates rather than single flips — the same point the sample-size chapter makes about treating one run as if it were the truth.

Same evaluation on both sides

A comparison is only fair if both sides face the identical evaluation — the same dataset, the same split, the same scoring rule. Score A on one set and B on another and the difference could be the datasets, not the systems. This sounds obvious but slips in subtly: comparing today's config on today's data against last month's number on last month's data is not a comparison, because the data moved underneath it.

Look at the tasks that changed, not the delta

Here the one-change discipline pays a second dividend, and it's easy to miss. Because A and B face the same tasks and differ in exactly one knob, the comparison is paired — and tightly so. Most tasks will land identically on both sides; the knob didn't touch them. The tasks whose outcome changed are where the knob's effect can be, and that set — not the delta — is your result. It is also, per the caveat above, where run-to-run noise shows up, which is precisely why it's the set worth examining rather than summarising.

So don't read the comparison off the delta. Read it off two counts:

• Fixed — tasks A failed and B passes.
• Broke — tasks A passed and B fails.

Everything else cancels. A change that fixes 12 and breaks 2 is a real improvement you can defend; one that fixes 5 and breaks 4 is churn wearing a +1-point costume. The sample-size chapter has the machinery — Wilson intervals for absolute accuracy, exact McNemar on the fixed/broke split, and why interval overlap is not a test — so use it there rather than re-deriving it per comparison. What matters here is the habit: every comparison produces a list of changed tasks, and you read them.

A zero delta is not "no effect"

The sharpest consequence: two configurations can score exactly the same and still behave differently, because a change that fixes some tasks and breaks an equal number nets out to zero. The aggregate says "no effect." The paired view says "this knob rewrote outcomes on tasks you should look at." Only one of those is true.

This matters most for changes you're inclined to ship because they seem harmless. A prompt tweak that leaves accuracy flat has not been shown to be safe — it has been shown to be flat, which is a much weaker claim.

Before concluding that a small fixed/broke split is real, though, check whether it could be noise. Some knobs are deterministic given the model's output — a post-processing rule, a validator, a repair step — and for those, a flip is unambiguously the knob. Others require a fresh generation on each side, and there a 1-fixed/1-broke result may be nothing at all. Re-run before you interpret.

When the comparison can't resolve

Sometimes the changed-task list is too small to conclude anything — a knob that fixes 2 and breaks 0 is suggestive but, as the sample-size chapter shows, can't reach significance at all. That's a real outcome, and the honest response is one of three moves: gather more data, decide on secondary grounds (simplicity, cost, latency), or read the changed traces and make a judgement you label as a judgement.

Mental model

Iteration is a stream of A-vs-B comparisons. Make each one interpretable by changing exactly one part of the configuration and holding the rest fixed; make it fair by scoring both sides on the same current evaluation; then read the result off the tasks whose outcome changed — what the knob fixed and what it broke — not off the difference between two percentages. One change, same eval, and a list of changed tasks you actually look at.

Common mistakes

• Changing several things at once. The score moves but you can't attribute it — motion without knowledge.
• Comparing against a remembered number. The dataset or scoring may have drifted; run both sides through the same current eval.
• Judging by the delta. Two percentages subtracted throw away which tasks moved and which way; fixed and broke are the actual result.
• Reading a flat score as "safe." A zero delta can hide a knob that broke as many tasks as it fixed.
• Treating every flip as caused by the knob. With one attempt per side, a stochastic model can flip a task on its own; re-run before interpreting a small fixed/broke split.
• Comparing error bars instead of testing the pair. Overlap isn't a test; it turns real differences into ties.
• Different scoring or splits per side. Any difference in the evaluation itself contaminates the comparison.

Practical guidance

• Change one knob per comparison and hold every other part of the configuration fixed, so the change in outcomes is attributable.
• Run both configs through the same current evaluation back to back; never compare to a historical number.
• Report every comparison as fixed / broke / unchanged, and test that split with the paired method in Sample size and uncertainty.
• Read the broken tasks before shipping — a net-positive change can still be unacceptable if what it broke matters more than what it fixed.
• Repeat both configurations when the knob requires a fresh generation and the changed-task list is small; compare paired outcome rates rather than single flips.
• When the changed-task list is too small to resolve, say so and decide on secondary criteria, rather than promoting a delta that can't reach significance.
• Record each comparison as (what changed, on what eval, fixed, broke, is-it-significant) so the iteration history is auditable.

Summary

• Improving a system is a sequence of controlled A-vs-B comparisons.
• Make each interpretable (change one thing, hold the rest fixed) and fair (same current evaluation on both sides).
• One change on the same tasks makes the comparison paired: the evidence is the tasks whose outcome changed, and the delta is a lossy summary of them.
• Report fixed and broke, test the split, and read what broke — a zero delta is not "no effect."
• Holding the rest fixed rules out other configuration changes as the cause, not chance; where a knob needs a fresh generation, repeat both sides before believing a small split.
• This project's guardrail knob shows both ends: +23 / −1 on qwen3b, and +1 / −1 behind a perfectly flat score on qwen7b — and because the correction post-processes the same generation, those flips are the knob, not noise.

Knowledge check

You switch to a new prompt AND a bigger model at once, and accuracy jumps 6 points. Your teammate credits the new prompt. What's the problem, and how should you have run it?

You changed two things, so the 6-point gain isn't attributable to either — it could be the prompt, the model, or an interaction, and crediting the prompt is a guess. To attribute it, run controlled comparisons that change one knob at a time: same model, old vs new prompt (isolates the prompt); then fixed prompt, old vs new model (isolates the model). Each delta, measured on the same evaluation and checked against the noise, tells you what actually caused the improvement.

A new prompt scores 91% vs the old prompt's 89% on a 60-example set. Ship it?

You can't tell yet, and the reason is that the percentages don't contain the answer. On 60 tasks one task is worth 1.7 points, so "+2 points" is a net movement of about one task — and a net of one is consistent with wildly different realities: the prompt fixed 1 and broke 0, or it fixed 6 and broke 5. Those demand opposite decisions, and the two scores cannot distinguish them.

So the first move isn't a statistical test, it's a query: which tasks changed outcome, and in which direction? Then:

• Fixed 1, broke 0 — a real but unresolvable signal. One discordant task can't reach significance (you'd need at least six even for a clean sweep), so this is a tie on the evidence. Decide on simplicity, cost, or latency.
• Fixed 6, broke 5 — the prompt is not a small improvement; it is a substantial behavioural change that nearly cancels. Read all eleven traces. The five it broke may matter more than the six it fixed.

Either way, "+2 points" was never the finding. And note the wrong route to the right answer: "the confidence intervals overlap, so it's a tie" would land on tie here by luck, using a rule that also calls decisive results ties.

You add a validation retry to your pipeline. Accuracy before: 82%. Accuracy after: 82%. Your teammate concludes the retry is useless and wants to remove it to save latency. What have you actually learned?

That accuracy is flat — which is not the same as the retry having no effect. Build the paired table before deciding. If the retry changed no outcomes at all, the teammate is right and the latency is being spent for nothing. But if it fixed 4 tasks and broke 4, the retry is rewriting behaviour and the flat score is a coincidence of arithmetic — removing it would give back 4 tasks and take back 4 different ones, which is a change, not a no-op.

A retry deserves one extra step, though: it issues a fresh generation, so an outcome that changed may have changed by chance rather than because of the retry. With one attempt per side you cannot tell those apart. Re-run both configurations and compare paired outcome rates before concluding that a 4/4 split is the retry's doing — a retry is exactly the kind of knob whose whole mechanism is re-rolling a stochastic step.

This project's guardrail on qwen7b is the deterministic counterpart: accuracy moves by 0.0 points while 1 task is fixed and 1 broken, and because the correction post-processes the same generation, those two flips are certainly the knob. Either way the lesson holds — a zero delta is a statement about a sum, not about behaviour, and "no measurable improvement" is a much narrower claim than "does nothing."

Related chapters

• The model is only one component — the freeze-the-configuration discipline comparisons rely on
• Sample size and uncertainty — deciding whether a comparison's gap is real
• Champion–challenger testing — structured comparisons for promoting a config
• Release gates — where a winning comparison must still clear the bar