Evaltudepreview

Part 3 — Evaluation Foundations

Evaluating complete pipelines

Valid is not correct·Evaluation·8 min read

Real systems are multi-stage — retrieve, prompt, model, parse, validate, correct, execute — and a single end-to-end score hides how and where the pipeline diverged. This lesson closes the evaluation foundations by scoring the whole pipeline both ways: end-to-end for the user outcome, and stage-by-stage to narrow the investigation.

The chapters so far scored an output against an answer. But an LLM application is not one output — it is a pipeline of stages, and a failure at any one of them shows up as "the answer was wrong." A single end-to-end number tells you whether the user got the right outcome; it hides how and where the pipeline diverged. Evaluating a complete pipeline means measuring it two ways at once: end-to-end for what the user experienced, and stage-by-stage to narrow where to investigate. This closes the evaluation foundations and opens the door to failure analysis.

What you will understand by the end

  • Why a single end-to-end score hides how and where a multi-stage system diverged.
  • The difference between end-to-end and stage-level evaluation, and why you need both.
  • How the raw-vs-corrected split you already met is a two-stage pipeline view.
  • How this connects evaluation foundations to failure analysis.

An LLM system is a pipeline of scorable stages

Recall the anatomy: a request flows through retrieval, prompt assembly, the model, parsing, validation, a correction policy, and execution. Each stage takes an input and produces an output. That creates measurement points, but not every stage is equally easy to score:

   retrieve ─▶ prompt ─▶ model ─▶ parse ─▶ validate ─▶ correct ─▶ execute ─▶ answer
      │           │        │        │          │           │          │
   recall?    complete?  intent  parses?   schema-ok?   fixed vs   query runs?
   right docs?           right?             (syntax)    broke it?  right rows?
      └──────── stage-level metrics narrow where to investigate ─────────┘
   └────────────────────── end-to-end metric: did the user get the right answer? ─────────┘
Stage Useful measurement Qualification
Retrieval Recall@k; relevant-evidence coverage Requires annotated relevant evidence
Prompt assembly Required-context and rule coverage Requires a prompt contract
Model Semantic accuracy Interpret conditionally when context may be defective
Parsing Parse success; false rejection A valid output can still be rejected by the parser
Validation Acceptance and rejection accuracy Schema validity is not semantic correctness
Correction Touched, fixed, broke, changed-incorrect, unchanged Effect is configuration- and dataset-specific
Execution Execution success; result correctness A query can run successfully and still be wrong
Key idea

An end-to-end score answers "did it work?"; stage-level scores narrow where to investigate. You need both. End-to-end is the outcome the user actually got; stage-level evidence turns a red number into a tractable search. Reporting only end-to-end is like a smoke alarm with no location — you know there's a fire, not which area to inspect first.

The eligible population belongs beside every stage rate. For a policy stage, intervention rate is the number of touched trials divided by all eligible trials; intervention precision is fixes divided by touched trials; and net accuracy effect is fixes minus regressions, divided by all evaluated trials. Execution success might use all requests or only requests that reached execution. Model semantic accuracy is hard to interpret when retrieval supplied the wrong evidence. A rate without its eligible population can make one stage look healthier than it is and make two systems incomparable.

Why end-to-end alone misleads

Two systems can share the same end-to-end accuracy for completely different reasons — one has a weak retriever and a strong model, the other the reverse — and you'd treat them identically if all you saw was the final number. Worse, a stage can mask another: a good correction policy can hide a weak model, and a lenient final check can hide upstream damage. Per-stage measurement makes these different failure profiles visible and narrows the investigation; it does not by itself establish which stage caused them.

Watch out

An empty or wrong final answer is not automatically "the model was wrong." It might be a retrieval miss (nothing relevant fetched), a parsing bug (good output dropped), or a schema that under-constrained. Diagnosing a pipeline starts by identifying where the defect first became visible, then testing where it originated. This is the core habit of failure analysis.

A red stage metric identifies where an error became visible, not necessarily where it originated. A retrieval miss can make retrieval recall, model accuracy, groundedness, and final correctness all fail on the same trial. To attribute root cause, inspect the trace, condition downstream metrics on valid upstream inputs, or rerun the downstream stage with a known-correct upstream artifact. If the model succeeds with oracle context, that controlled rerun isolates the retriever more convincingly than four simultaneously red metrics do.

The raw-vs-corrected split is a two-stage view

You have already seen the smallest version of pipeline evaluation. In The model is only one component, the dashboard labels the two views raw model accuracy and corrected system accuracy. Operationally, this comparison scores the validated pre-policy output and the post-policy output. The first label is convenient shorthand for the system output before correction, not a measurement of isolated model capability. The gap is the observed net intervention effect in that configuration and on that dataset: fixes minus regressions, not an independent contribution owned by the policy.

Observed evidence

This project measures both stages, which is the only reason its results are interpretable: scoring before and after correction reveals the policy's observed net effect — something the end-to-end number alone would hide. The same policy substantially helps the 3B configuration and nets to zero on the 7B configuration. On 7B it touched four trials: one fix, one regression, and two wrong-to-different-wrong changes. The flat aggregate therefore concealed meaningful behavioral changes, including one newly broken task. The result belongs to the model-policy-dataset interaction, not to a fixed amount of value inherent in the correction stage. The guardrail intervention analysis shows the trial-level accounting. See the correction stage's trials →

Mental model

Evaluate a pipeline at two levels. Measure end-to-end outcomes to determine whether users got the right result, and measure intermediate stages to narrow the source of failures. Because errors propagate and stages interact, stage metrics are diagnostic evidence — not automatic causal attribution. Confirm root cause with traces, conditional metrics, and controlled reruns.

Common mistakes

  • Only measuring end-to-end. You learn that it failed, not how or where the defect first became visible; every fix becomes a guess.
  • Letting a stage mask another. A strong correction policy or lenient final check hides an upstream weakness unless you measure stages separately.
  • Treating a moved stage metric as causal attribution. It tells you where the outcome changed; use traces, conditioning, and controlled reruns to establish why.
  • Hiding the denominator. A parse rate over all requests and one over parse-eligible responses are different claims; state which population each rate describes.

Practical guidance

  • Instrument each stage's input and output, and define a metric plus its eligible denominator per stage alongside the end-to-end score.
  • Set outcome gates for user-visible correctness. Enforce system invariants such as authorization, data isolation, and schema requirements. Track operational SLOs such as latency and availability. Set explicit minimums for important slices. These are separate requirements, not one interchangeable threshold.
  • Use stage metrics and traces to narrow the investigation. To verify that an intended modification caused an improvement, run a controlled comparison that holds the other relevant factors fixed.
  • For any policy stage, report trials touched, incorrect → correct, correct → incorrect, incorrect → different incorrect, unchanged, and net end-to-end delta. The pre/post gap is a scoped net intervention effect, including when it is negative.
  • Feed stage-level failures into failure analysis — locating where a defect became visible narrows the investigation; it does not alone identify the originating cause.

Summary

  • LLM systems are pipelines with stage-level measurement points; not every stage is directly scorable without a contract, reference evidence, or a conditional denominator.
  • Measure both end-to-end user outcomes and stage-level diagnostic evidence; gate critical invariants and operational requirements too.
  • The raw-vs-corrected gap is a configuration-specific net intervention effect, not an independently attributable stage contribution.
  • Stage metrics narrow the investigation; traces and controlled reruns establish root cause.

Knowledge check

Two RAG systems both score 78% end-to-end. Why might you still strongly prefer one, and what would reveal the difference?

You cannot choose from 78% alone. Stage metrics can reveal different failure profiles, ceilings, and repair costs — for example, strong generation under oracle context paired with a replaceable weak retriever. But preference also depends on failure severity, production distribution, repair effort, latency, and cost. The stage profile informs the engineering decision; it does not determine it.

Your system's end-to-end accuracy is 88%, and a teammate credits "the model." How does the raw-vs-corrected view test that claim?

Score the validated pre-policy output and the post-policy output, then inspect what the policy touched. If the first is 63% and the second is 88%, the policy has a +25-point observed net intervention effect for this model, dataset, and configuration. Report fixes and regressions underneath that net. The comparison disproves the claim that the final 88% describes the model alone, but it does not assign an interaction-free 25-point contribution to the policy in every system.

Related chapters