Release gates

A regression suite that runs on every change but never stops anything is a smoke alarm with the wires cut. The piece that gives it teeth is the release gate: a rule that says a change may ship only if it clears an agreed bar on the metrics that matter. The gate is what converts an evaluation number from something you glance at into a decision that blocks — and designing it well (which metrics, what threshold, agreed by whom, when) is what keeps a continuously-changing system from degrading one "small" merge at a time.

What you will understand by the end

• What a release gate is and why a suite without one doesn't protect you.
• Which metrics to gate on — and why correctness, not validity, must be among them.
• How to set a threshold that blocks regressions without blocking all progress.
• Why the gate must be agreed in advance, not negotiated at merge time.

A gate is a suite with authority

A release gate is a pre-committed rule: this change ships only if the evaluation clears these thresholds. It runs the regression suite (and any other required checks) automatically on the change, and a failure blocks the release rather than merely warning. That authority is the whole point — without it, evaluation is advisory, and advisory checks get overridden by whoever is in a hurry.

Gate on the metrics that matter — correctness included

What you gate on determines what the gate protects. The cardinal rule from Part 4 applies with full force: a validity check ("100% schema-valid") is not a quality gate — it can pass while correctness collapses. So a real gate includes a semantic correctness threshold on the held-out set, and typically:

• Correctness on held-out data — the primary bar, measured with a meaning-level scoring method.
• No regression on the regression suite — previously-passing cases must still pass.
• Guardrail metrics — things that must not get worse even if the headline improves (latency, cost, a critical segment's accuracy).

Set the threshold on regressions, not on the delta

A gate threshold has to block genuine regressions without blocking every change over statistical noise. Two forces:

• Set it too tight and normal sampling variation fails good changes.
• Set it too loose and real degradation slips through.

The tempting resolution is "block if the drop exceeds the noise." Don't build the gate that way. A gate is asking a regression question — did this change break anything that used to work? — and that is a paired question about specific tasks, not a question about the gap between two percentages. The change and the champion run the same suite, so the gate can name the tasks that changed hands: how many the change fixed, and how many it broke. A candidate that fixes 6 and breaks 5 has a flattering aggregate and five broken behaviours; a delta-based gate waves it through.

So a gate has three separate bars, and only the first is a threshold on a score:

• An absolute bar. Held-out correctness must clear an agreed level — stated honestly as the lower end of its Wilson interval, so the gate isn't passed by an optimistic point estimate. Write the policy out in full: "the one-sided 95% Wilson lower bound on held-out correctness must be ≥ 90%." Confidence level and one- versus two-sided both change where the bar actually sits, so leaving them implicit means two people can read the same gate differently. This is a deliberately conservative acceptance policy, not the uniquely correct statistical gate — a team with a large suite and a costly false block might reasonably gate on the point estimate instead. It also assumes your held-out set is a representative sample of the workload you care about; if it isn't, the interval is precise about the wrong population.
• A regression bar. A count of confirmed broken tasks against the current champion, with an explicit tolerance agreed in advance — often zero for protected cases.
• Guardrail bars. Latency, cost, and critical-segment accuracy that must not worsen, each with its own stated allowance.

"Not significant" is not a reason to ship a regression

This is the trap that a statistics-shaped gate walks straight into. Suppose the candidate breaks three previously-passing tasks and fixes none. On 88 tasks that fails to reach significance — a 3–0 split is well short of what a paired test would call conclusive. A gate keyed to statistical significance therefore passes it.

But the p-value was never the right question to ask about those three tasks. It answers "did the aggregate move?" when what you need to know is "do these three cases work?" — and that second question is answered by looking at them, not by testing a population.

First, ask why the case flipped

Which does not mean every observed flip should block. A gate that blocks on any changed outcome becomes exactly the twitchy gate this chapter warns against, because a single hosted-model attempt can flip on its own. Before a break blocks the release, classify it:

• A protected deterministic case — a fixed input whose correct output is unambiguous and which fails identically on every repeat. One failure is enough to block; there is nothing statistical about it.
• A stochastic model outcome — a case that could plausibly flip between runs of the same configuration. Don't block on a single observation and don't wave it through either: re-run it, on both the champion and the candidate. Better, gate these on a pre-declared pass rate ("this case must pass 5 of 5") rather than on a single attempt, which converts a coin-flip into a decision.
• A scorer or label failure — the model's answer is fine and the check is wrong, or the expected output was mislabelled. Blaming the candidate here trains the team to distrust the gate. Fix the suite, and treat the finding as valuable.

Doing this triage is cheap — it's a handful of re-runs on a handful of tasks — and it is what makes the blocking rule credible. A gate that blocks on confirmed breakage gets respected; one that blocks on sampling noise gets overridden, and an overridden gate protects nothing.

Agree it in advance

A gate negotiated at merge time isn't a gate — it's a debate the deadline usually wins. The threshold, the metrics, and who can grant an exception must be decided before the change is on the table, so the rule constrains the moment instead of bending to it. Exceptions should be rare, explicit, and logged, not a quiet override.

Mental model

A release gate is a pre-agreed rule that blocks a change unless it clears thresholds on the metrics that matter — an absolute correctness bar on held-out data, a cap on tasks the change breaks against the champion, and guardrail metrics that mustn't worsen. It gives evaluation authority (block, not warn), decides on the tasks that changed hands rather than the delta, and is agreed in advance so the deadline can't renegotiate it.

Common mistakes

• A suite with no gate. Checks that only warn get ignored; without blocking authority quality erodes merge by merge.
• Gating on validity or aggregate alone. Both can pass while correctness or critical cases regress; gate on semantic correctness plus guardrail metrics.
• Gating on the delta. A candidate that fixes 6 and breaks 5 shows a positive net and six broken behaviours; gate on the broken count.
• Treating "not significant" as permission. A p-value can't un-break a case that used to pass; protected regressions block regardless of sample size.
• Deriving the tolerance from statistics. How much breakage you can absorb is a product decision, not an output of a test.
• Negotiating the gate at merge time. A bar decided under deadline pressure isn't a bar; agree it in advance.

Practical guidance

• Make the gate block, not warn, and run it automatically on every change.
• Gate on held-out correctness, a regression check against the champion, and guardrail metrics that must not worsen — never validity alone.
• State the absolute bar against the lower end of the confidence interval, so an optimistic point estimate can't clear it.
• Gate the regression check on the count of broken tasks, with a pre-agreed tolerance — typically zero on protected cases.
• Make the gate name the tasks it blocked on, so the failure is a list to read rather than a number to argue with.
• Pre-agree the metrics, thresholds, and exception process; log the rare exceptions instead of quietly overriding.

Summary

• A release gate gives the regression suite authority: a change ships only if it clears the bar.
• Gate on correctness (held-out), no regression, and guardrail metrics — validity alone is not a quality gate.
• A gate asks a paired regression question — what did this change break? — so it decides on broken-task counts, not on the difference between two percentages.
• The tolerance is an explicit product decision agreed in advance; "not statistically significant" is not a reason to ship a regression.
• This project's "which systems clear the bar" framing is a release gate in action.

Knowledge check

A team gates releases on "100% schema validity and no crashes." A change ships that passes both but drops semantic accuracy from 92% to 78%. What was wrong with the gate?

It gated on structural health (validity, no crashes) but not on semantic correctness — and those are independent axes, so a change can keep 100% valid JSON while its answers become far more often wrong. The gate needed a held-out correctness threshold (with a meaning-level scoring method) as its primary bar, plus a regression-suite pass. Validity is a syntax check; a quality gate must measure meaning, or it waves through exactly this kind of degradation.

On an 88-task suite a candidate scores 1.1 points above the champion. The paired breakdown shows it fixed 6 previously-failing tasks and broke 5 previously-passing ones, two of which are protected deterministic cases that fail identically on every repeat. A McNemar test on 6-vs-5 is nowhere near significant. Does your gate pass it?

No — and notice that every aggregate signal here says yes. The score went up, and the statistics say the change is indistinguishable from no change. A gate keyed to either one ships it.

The gate should block on the two broken protected cases, full stop. Those are not a hypothesis about a population; they are two behaviours that used to be correct and are now wrong, and no p-value restores them. "Not significant" means the aggregate movement could be chance — it says nothing about whether these specific cases work.

It's also worth seeing what this candidate actually is. Eleven of 88 tasks changed hands for a net of one — which is the +1.1 points, since one task on 88 is 1.14 points. That is not a small improvement; it's a substantial behavioural change with a near-cancelling score, and the headline delta is the least informative fact available about it. The right response is to read all eleven traces.

Note the qualifier in the question: the two protected cases fail on every repeat. That is what licenses blocking on them outright. Had they been ordinary stochastic outcomes that flipped once, the correct first move would be to re-run them rather than to block — see the distinction above.

Then what is confidence-interval reasoning for, if the gate doesn't block on statistical significance?

For the absolute bar, not the regression bar — and for resisting over-reading aggregate movement.

Stating the absolute bar as "held-out correctness ≥ 90%" invites a system measured at 80/88 — 90.9% — to clear it on 88 tasks, when the honest range around that estimate reaches well down into the 80s: its one-sided 95% Wilson lower bound is 84.6%, and the lower end of the two-sided 95% interval is 83.1%. Requiring the lower bound to clear the bar makes the gate's confidence explicit and makes small datasets harder to pass with, which is the right incentive.

But notice you had to pick two things to get a number at all — the confidence level, and one-sided versus two-sided — and they move the bar by 1.5 points here. Write both into the policy. And be clear about what kind of choice it is: this is a conservative acceptance rule, not the one statistically correct gate. It deliberately puts the burden of proof on the candidate, which is right when shipping a regression is expensive and wrong when blocking a good change is.

The two roles then divide cleanly. Uncertainty stops you believing a 1-point aggregate wobble means anything, and sets an absolute bar you can't pass by luck. The list of confirmed broken tasks and a pre-agreed tolerance decide whether to ship. Using significance for that second job is what lets real regressions through.

Related chapters

• Regression suites — the suite the gate enforces
• Why valid JSON is not semantic correctness — why the gate must include correctness
• Sample size and uncertainty — setting a threshold that reacts to real change
• Champion–challenger testing — the gate applied to promoting a new config