Part 9 — Continuous Evaluation

Online experiments

Comparing and evolving·Evaluation·6 min read

An offline win is a prediction; an online experiment is the confirmation. This chapter is about testing a change on real traffic — randomised A/B assignment, a pre-chosen success metric, guardrail metrics, and enough data for significance — so that 'better in the lab' becomes 'better for users,' which is the only claim that ships.

Offline evaluation, for all its discipline, measures your system on a dataset — a model of reality. The online experiment measures it on reality itself: real users, real requests, real outcomes. It is slower, riskier, and noisier than offline, and it is the only thing that can confirm an offline win actually helps users. This chapter is how to run one honestly — because an online test done carelessly produces confident numbers that are just as misleading as a bad offline set.

What you will understand by the end

  • Why an offline win needs online confirmation, and what online uniquely measures.
  • The anatomy of an A/B test: randomised assignment, a success metric, guardrail metrics.
  • The traps — confounders, peeking, insufficient data — that fake a result.
  • When online is worth its cost, and when offline is enough.

Offline predicts, online confirms

An offline number says "on our test set, B beats A." That's a prediction about production, trustworthy only to the degree the test set is representative. The online experiment tests the prediction against the real distribution, and it can measure things offline can't: actual user behaviour (do they accept the answer, edit it, retry, convert), the true input mix, and effects that only appear at scale. When offline and online disagree, the gap is information — usually that the offline set missed something real.

Key idea

Offline evaluation gates cheaply and reproducibly; the online experiment is the reality check that a gated win holds for real users. You filter with offline (fast, safe) and confirm with online (real, costly) — neither replaces the other, and shipping a change on offline evidence alone is shipping on a prediction you never verified.

Anatomy of an A/B test

A sound online experiment has a few non-negotiable parts:

  • Randomised assignment. Users (or requests) are split at random between the current system (A) and the change (B), so the two groups differ only by the change — the online version of "hold everything else fixed."
  • A pre-chosen success metric. The one outcome that defines "better" — task success rate, user acceptance, conversion — decided before the test, not fished for after.
  • Guardrail metrics. Things that must not get worse even if the success metric improves — latency, cost, error rate, a critical segment's experience.
  • Enough data for significance. Real traffic is noisy; you need enough samples for the difference to clear the uncertainty, and you decide the sample size before you start.
Watch out

Two traps fake online results. Confounders: if the groups differ by anything besides the change — a concurrent launch, a time-of-day skew, a bad randomisation — the difference isn't caused by your change. Peeking: repeatedly checking an A/B test and stopping the moment it looks significant massively inflates false positives; fix the sample size (or use a sequential-testing method) in advance and don't stop early on a favourable wobble.

The costs are real

Online experiments are expensive in ways offline isn't: they're slow (you wait for enough traffic), risky (group B is a real, possibly worse experience for real users), and confounded (the world changes during the test). So you don't online-test everything — you online-test the changes that won offline and matter enough to justify the cost and risk. A small, low-stakes change vetted offline can often ship without an A/B test; a major change to a high-traffic path should be confirmed online before full rollout.

Observed evidence

This project is deliberately offline — a fixed 88-task set, no live traffic — so its numbers are predictions awaiting exactly this online confirmation. That's the honest boundary: the 95.5% says "this configuration should be the default on this task," and an online experiment on real business questions is what would confirm the offline ranking survives contact with real users. Offline chose the challenger; online would ratify it. The offline prediction awaiting online confirmation →

Mental model

Offline predicts, online confirms. An A/B test measures a change on real traffic via randomised assignment, a pre-chosen success metric, and guardrail metrics that mustn't worsen — with enough data to clear the noise. Beware confounders and peeking. Online-test the changes that won offline and matter enough to justify the cost.

Common mistakes

  • Shipping on offline evidence alone. An offline win is a prediction; for high-stakes changes, confirm it on real traffic.
  • Choosing the success metric after the fact. Fishing for a metric that looks good inflates false positives; pre-commit it.
  • Peeking and stopping early. Repeated checks with early stopping fake significance; fix the sample size (or use sequential methods) in advance.
  • Ignoring confounders. Any group difference besides the change — concurrent launches, timing, bad randomisation — invalidates the result.

Practical guidance

  • Randomise assignment cleanly, pre-commit a single success metric and guardrail metrics, and compute the required sample size before starting.
  • Don't peek-and-stop; run to the planned sample (or use a proper sequential test) and read significance against the uncertainty.
  • Reserve online tests for changes that won offline and matter enough to justify the cost and user risk; ship low-stakes offline-vetted changes directly.
  • Treat an offline/online disagreement as a signal to fix your offline set's representativeness, not as noise.

Summary

  • The online experiment confirms on real users what offline only predicted; it measures behaviour and the true input mix offline can't.
  • A sound A/B test needs randomised assignment, a pre-chosen success metric, guardrail metrics, and enough data.
  • Guard against confounders and peeking, the two classic ways to fake an online result.
  • Online is slow, risky, and costly, so reserve it for offline-winning changes that matter — this project's offline 95.5% is exactly a prediction awaiting online ratification.

Knowledge check

An A/B test's success metric crosses significance on day 2, so the team stops and ships B. Why is that dangerous?

This is peeking with early stopping: if you repeatedly check a running experiment and stop the moment it looks significant, you dramatically inflate the false-positive rate — random fluctuations will cross the threshold by chance long before the real effect is established, so the "win" may be noise. The fix is to compute the required sample size in advance and run to it (or use a proper sequential-testing method designed for continuous monitoring), reading significance only at the planned stopping point rather than the first favourable wobble.

Your change wins clearly offline. Give one reason you'd still run an online test, and one reason you might skip it.

Run it if the change is high-stakes on a high-traffic path: the offline win is a prediction that depends on your test set being representative, and only real traffic confirms it helps actual users (and that guardrail metrics like latency/cost didn't worsen) before full rollout. Skip it if the change is small and low-risk and was solidly vetted offline — the cost, delay, and user risk of an A/B test aren't justified for a minor change whose offline evidence is already convincing.

Related chapters