Versioning a complete system configuration

Part 3 began by insisting the unit of everything is the system — a versioned configuration, not a model. This closing chapter takes that seriously as an engineering practice: if a system is a configuration, then pinning that configuration is what makes a result reproducible, a comparison fair, and a regression traceable. A number attached to a system you can't reconstruct is folklore. Versioning the complete configuration — treating it as one artifact, like code — is what turns the whole discipline of this book from advice into something you can actually operate.

What you will understand by the end

• Why every result must be attached to a reproducible configuration, not "the model."
• Everything the configuration includes — and why a missing piece breaks reproducibility.
• Why config-as-code underpins comparisons, gates, and regression suites.
• The special problem a hosted model poses for versioning.

A result belongs to a configuration you can reconstruct

Recall the seven-or-so parts that make up a system: model, prompt, output schema, decoding parameters, provider settings, parser, validator, correction policy — plus, for extended systems, the retriever/index and the tool set or harness. A measured number is a property of all of them, frozen together. So to reproduce or defend that number, you must be able to reconstruct that exact set. If any part is unrecorded — "we changed the prompt sometime around then" — the result is no longer attributable to anything.

Version the whole thing, as one artifact

The discipline is to capture the entire configuration as a single versioned manifest — not just the model id, but every part that can move the number:

   system configuration v1.5
   ├── model            id + version (or pinned weights)
   ├── prompt           template + assembly logic (versioned)
   ├── schema           output shape + validation rules
   ├── decoding         temperature, top-p, max tokens, seed
   ├── provider         endpoint / serving settings
   ├── parser+validator versions
   ├── correction policy version
   └── retriever/tools  index snapshot, tool set (if any)

Treat this manifest like code: it's committed, diffable, and every change produces a new version. Then a "change" is never ambiguous — you can see exactly which part moved between v1.4 and v1.5, which is the precondition for attributable comparison.

Config-as-code makes the whole program work

Versioning isn't bookkeeping for its own sake — it's the substrate the rest of the book stands on:

• Comparisons require holding every part fixed but one; you can only do that if every part is pinned.
• Champion–challenger needs a named champion configuration to promote against and fall back to.
• Regression suites and gates run against a specific version, and a caught regression is traced by diffing configurations.
• Reproducing a failure requires reconstructing the exact system that produced it.

Every one of those becomes possible only because the configuration is versioned. Config-as-code is the quiet foundation under continuous evaluation.

The hosted-model wrinkle

Hosted models complicate versioning, because the weights live on someone else's servers and can change under you. You can pin a model identifier, but you don't control whether that version stays available or behaves identically over time — so full reproducibility may be outside your reach. The mitigations: record exact model/version identifiers, snapshot representative outputs, and treat a hosted-model change as a configuration change that must re-clear the gate. Self-hosting is the only way to truly pin the weights.

Mental model

A system is a configuration, so a result is only as reproducible as your ability to reconstruct that configuration. Version the whole thing — model, prompt, schema, decoding, provider, parser, policy, retriever/tools — as one code-like artifact. That versioning is the substrate under comparisons, gates, regression suites, and root-cause; without it, every number is folklore. Hosted models can drift, so pin what you can and treat a change as a new version.

Common mistakes

• Labelling results with a model name. The number belongs to the whole configuration; the model is one part.
• Leaving components unversioned. A silent decoding, reindex, or provider change moves the number with nothing to attribute it to.
• Not treating config as code. Un-committed, un-diffable configuration makes comparisons and regression tracing impossible.
• Assuming a hosted model is stable. It can change under you; record identifiers and re-gate on provider changes.

Practical guidance

• Capture the entire configuration as one versioned, committed manifest — every part that can move the number, not just the model id.
• Treat configuration as code: diffable, reviewed, and re-versioned on every change, so a "change" is always attributable to a specific part.
• Attach the version to every reported result, so any number can be reconstructed and defended.
• For hosted models, pin version identifiers, snapshot outputs, and re-clear the gate whenever the provider's model changes.

Summary

• A result belongs to a reproducible configuration — all seven-or-so parts (plus retriever/tools) frozen together, not "the model."
• Version the whole configuration as one code-like artifact, so every change is attributable.
• Config-as-code is the substrate under comparisons, champion-challenger, regression suites, gates, and root-cause.
• Hosted models can drift, limiting reproducibility — pin what you can and treat a provider change as a new version. This project's versioned rows are why its comparison is defensible.

Knowledge check

Six months after reporting a 94% result, you can't reproduce it — same model, same prompt file. What likely changed, and what should versioning have captured?

Something outside the obvious knobs moved. Likely culprits: a hosted model updated under you (same id, different behaviour), a retriever reindex or changed document set, a provider default (decoding temperature, safety settings) that shifted, or an unrecorded change to decoding parameters. Versioning should have captured the complete configuration — exact model/version identifier, decoding params (incl. seed), provider settings, retriever index snapshot, parser/validator/policy versions — so the exact system could be reconstructed. With hosted models, it should also have snapshotted representative outputs, since true weight-level pinning isn't possible.

Why is versioning the configuration a prerequisite for controlled comparisons and regression suites, not just good hygiene?

Because both depend on knowing exactly what differs. A controlled comparison isolates one change by holding every other part fixed — impossible unless every part is pinned. A regression suite traces a caught failure to a specific change by diffing the configuration between versions — impossible if the configuration isn't captured as a diffable artifact. Champion-challenger needs a named champion to promote against and revert to. So config-as-code isn't just tidiness; it's the substrate that makes attributable comparison, traceable regression, and reproducible root-cause possible at all.

Related chapters

• The model is only one component — the configuration this chapter versions
• Prompt and model comparisons — why every part must be pinned to attribute a change
• Hosted versus self-hosted systems — the model-drift problem for versioning
• Regression suites — run against, and traced by diffing, versioned configurations