Trial-level traces

Part 4 and Part 5 got you a trustworthy number. This part is about what to do when that number isn't 100% — which is always. And the first move is counterintuitive for a field obsessed with metrics: stop looking at the aggregate and open a single failing trial. A score summarises; it cannot explain. Explanation lives in the individual case — the exact input, what each stage produced, and where the expected and actual answers diverged.

What you will understand by the end

• Why the aggregate number can't tell you how or why the system fails.
• What a trial-level trace contains and why you need every stage of it.
• How reading failures one at a time drives every diagnosis and fix.
• Why this habit is the foundation the rest of Part 6 builds on.

The aggregate can't explain

"78% accuracy" tells you the system fails about one time in five. It does not tell you which inputs fail, whether they fail the same way, which stage broke, or whether the failures are one bug or twenty. Every one of those questions — the ones that actually lead to a fix — is invisible at the aggregate level and answerable only by looking at the cases underneath it.

What a trace contains

A useful trial-level trace records the whole journey of one request, so you can see not just that it failed but where:

• The input — the exact request, verbatim.
• Each stage's output — retrieved context, the assembled prompt, the raw model output, the parsed object, the validation result, any correction the policy applied, the executed result.
• Expected vs actual — the ground-truth answer beside what the system produced, with the difference made visible.
• The outcome and metadata — pass/fail, which split, which system configuration.

With all of that, a failure stops being "the system got it wrong" and becomes "the model picked top_n when the question wanted a breakdown, the validator accepted it because it was schema-valid, and no correction rule caught it" — a statement you can act on.

Reading failures is the work

There's no shortcut around actually reading the failing cases. Skim ten or twenty traces and patterns leap out that no aggregate reveals: the same boundary confusion again and again, a whole class of inputs the retriever misses, a correction rule firing when it shouldn't. Those patterns are the raw material for everything else in this part — taxonomies, confusion analysis, root cause. The engineers who improve LLM systems fastest are the ones who read the most failures.

Mental model

A score measures the size of the problem; a trial-level trace — input, every stage's output, expected vs actual — reveals its shape. You can't fix a number, only the specific failures it's made of, so failure analysis starts by opening individual traces and reading them until the patterns show. Every later technique in this part operates on what the traces reveal.

Common mistakes

• Trying to fix the aggregate. You can't act on "78%"; you act on the specific failing cases behind it.
• Traces without intermediate stages. Input-and-output only tells you it failed, not where — instrument every stage.
• Not actually reading failures. Patterns that drive fixes only appear when you read real traces, not summaries.
• Sampling only passes. Failures are where the information is; read the cases that went wrong.

Practical guidance

• Instrument every stage so each trace shows retrieval, prompt, raw output, parse, validation, correction, and result — not just the final answer.
• Put expected beside actual with the difference highlighted, so the divergence is obvious at a glance.
• Make traces easy to open and browse (filter by failed, by split, by system) — friction here is friction on every diagnosis.
• Read failures regularly, in batches; the patterns you spot become the input to the rest of Part 6.

Summary

• The aggregate measures the size of failure; only the trial-level trace reveals its shape.
• A useful trace shows the input, every stage's output, and expected vs actual — enough to localise where it broke.
• Reading failures one at a time is the source of every diagnosis and fix in this part.
• This project's openable, side-by-side trials are what turn a low score into an actionable cause.

Knowledge check

Your system scores 78% and you want to improve it. Why is "the accuracy is 78%" a useless starting point, and what do you do instead?

Because 78% tells you the size of the failure, not its shape — not which inputs fail, whether they fail the same way, or which stage broke, none of which you can fix directly. Instead, open the failing trials and read them: with each trace showing input, every stage's output, and expected vs actual, you'll see the patterns (a recurring boundary confusion, a retrieval gap, a misfiring rule) that name concrete, fixable problems. You fix the failures, not the number.

Why must a trace include intermediate stage outputs, not just the input and final answer?

Because a failure at any stage — retrieval, prompt, model, parsing, validation, correction — looks identical at the output: a wrong answer. Without the intermediate outputs you can't tell whether the retriever fetched nothing, the model misread the input, the parser dropped good output, or a correction rule broke it. The intermediate stages are what localise the failure to a specific component, which is the first step of every fix.

Related chapters

• Evaluating complete pipelines — the stages a trace records
• Aggregate metrics versus individual failures — why the average hides the cases
• Error taxonomies — organising what the traces reveal
• Root-cause analysis — turning a read trace into a fix