Diagnosing intent failures

With a schema, a dataset, and systems producing outputs, the case study turned to the question failure analysis can answer from traces: how do the failures present, and which patterns recur? This chapter is failure analysis run on the real experiment — opening the failing trials, sorting them into types, drawing the confusion, and narrowing the investigation to one semantic boundary. That diagnosis created a specific, testable intervention opportunity; the guardrail experiment that came next measured whether it worked.

What you will understand by the end

• How the case study's failures were read at the trial level.
• The error taxonomy they sorted into, and its dominant type.
• The confusion: ranking predicted where breakdown was expected.
• Why it's a semantic boundary failure, and what that implied.

Reading the failing trials

The diagnosis started where the failure-analysis workflow says it must: not with the aggregate, but with the individual failing trials. Each trace shows the validated pre-policy intent and the expected intent side by side, so a failure is legible — you can see which field diverged and how. Reading a batch of them, a pattern jumps out immediately, one that no aggregate accuracy could have revealed: the failures are not scattered, they cluster on one kind of mistake.

The taxonomy and its dominant type

The taxonomy and correction hypothesis were developed on the 20-task development split. The validated pre-policy 3B output was correct on 15 tasks and failed on 5. Each failure receives one primary category, so the taxonomy is mutually exclusive rather than multilabel.

Development failure category	Count	Share of 5 failures
Wrong pattern	5	100%
Wrong filter, with pattern, metric, and dimension correct	0	0%
Wrong metric or dimension, with pattern correct	0	0%
Other primary error	0	0%

The wrong-pattern count was directional, not a mixture of different swaps:

Expected → predicted pattern	Count
Breakdown → ranking	5
Ranking → breakdown	0
Other pattern swaps	0

The policy was shaped from this development evidence. The held-out and adversarial counts below are post-hoc descriptions of the completed evaluation, not evidence used to form the correction hypothesis. Across the full 88-task run there were 48 raw failures, assigned once each using this explicit precedence: pattern mismatch first; with the pattern correct, metric or dimension, then filter, then ordering or limit.

Primary raw failure category	Count	Share of 48 raw failures
Wrong pattern	27	56.3%
Wrong ordering or limit, with pattern, metric, dimension, and filters correct	16	33.3%
Wrong filter, with pattern, metric, and dimension correct	5	10.4%
Wrong metric or dimension, with pattern correct	0	0.0%

These categories are mutually exclusive by construction. Within the 27 wrong-pattern failures, 26 were the same breakdown → ranking swap. That is 96.3% of wrong-pattern failures and 54.2% of all 48 raw failures. The count supports calling it the dominant confusion; it does not imply that the remaining failures share the same cause or fix.

The confusion: ranking for breakdown

Drawing the confusion sharpens it further. The dominant off-diagonal cell is directional: the validated pre-policy output used top_n_by_metric (a ranking — top-N by some metric, descending, truncated) when the expected answer was metric_by_dimension_with_filter (a breakdown — group by the dimension, list all of them). The canonical instance is "Revenue by region last quarter":

   question: "Revenue by region last quarter"

   expected breakdown:
     pattern  metric_by_dimension_with_filter
     group    region
     sort     region ascending
     limit    100  (non-ranking output cap)

   predicted ranking:
     pattern  top_n_by_metric
     group    region
     sort     revenue descending
     limit    10   (top-N truncation)
              ▲ same aggregation, but unsupported ranking and truncation

The output pattern indicates that the system treated "by region" as sufficient evidence for ranking rather than a breakdown across regions. The underlying aggregation is the same, but the predicted intent adds unsupported ranking and truncation.

A semantic boundary failure

This is a textbook semantic boundary failure: two valid intent classes sit close together semantically, separated by explicit ranking language versus a plain grouping request, and the output falls on the wrong side. Crucially, it's valid but wrong — the intent is perfectly schema-legal, so schema validation cannot detect the mistake. Evaluation against the canonical intent reveals it, while a separately tested semantic policy may detect some instances at runtime. Naming the failure this precisely created a testable intervention: a boundary failure points at the exact distinction a prompt or policy can target. Whether that intervention generalises is a separate empirical question that must be re-measured on each configuration.

Mental model

Diagnosing the case study is failure analysis in practice: read the failing trials (not the aggregate), sort them into a taxonomy (dominant type: wrong pattern), draw the confusion (directional: ranking-for-breakdown), and name it a semantic boundary failure (valid but wrong, "by region" treated as ranking evidence). The concentration and direction created a testable, targeted intervention opportunity.

Common mistakes

• Diagnosing from the aggregate. "45.5%" hides that the failures cluster on one shape mistake; you must read the trials.
• Missing the direction. "Ranking and breakdown get confused" is weaker than "ranking predicted for breakdown" — direction specifies what an intervention would need to distinguish.
• Treating it as general weakness. One confusion accounts for 26 of 48 raw failures; the count identifies a priority without pretending the other 22 failures are the same problem.
• Forgetting it's valid-but-wrong. No structural check catches it; only measuring meaning against the expected intent does.

Practical guidance

• Start diagnosis by reading failing trials with expected beside actual, then type and count them to find the dominant class.
• Draw the confusion to get the direction — it identifies a specific distinction to test.
• Recognise boundary failures (valid but wrong, adjacent meanings) as opportunities for a high-leverage intervention; measure the result rather than assuming one fix clears the cluster.
• Let the diagnosis drive the next experiment — a precisely named failure makes a targeted guardrail test possible.

Summary

• The case study's failures were diagnosed by reading trials, typing them, and drawing the confusion — the failure-analysis toolkit on real data.
• The dominant failure is wrong pattern, specifically top_n_by_metric predicted where metric_by_dimension_with_filter was expected — ranking for breakdown.
• It's a semantic boundary failure: valid but wrong, with "by region" treated as sufficient ranking evidence.
• Its concentration and direction justified a targeted guardrail experiment, whose effect was then measured per configuration.

Knowledge check

On the full 88-task run, the validated pre-policy 3B configuration scores 45.5% (40/88). Why is reading the failing trials more useful than trying to improve the aggregate directly, and what did it reveal?

Because "45.5%" tells you the size of the problem, not its shape — you can't fix a number, only the specific failures behind it. Reading the failing trials (with expected beside actual) revealed that the failures concentrate on one type: the validated pre-policy output uses a ranking pattern (top_n_by_metric) where the question wanted a breakdown (metric_by_dimension_with_filter). On the full run, that directional swap accounts for 26 of 48 raw failures. This turns the aggregate into a specific intervention hypothesis. The later controlled pre-policy versus post-policy comparison measured a +25-point net effect on the 3B configuration; the diagnosis alone did not guarantee that result or its transfer to another system.

Why does the ranking-for-breakdown failure evade the schema's validity check, and why is it an opportunity for a high-leverage intervention?

It evades validity because the wrong intent is fully schema-legal — top_n_by_metric is a valid pattern with valid fields, so it parses and validates perfectly; it's just the wrong pattern for the question. Schema validation cannot detect that semantic mismatch. Evaluation against the canonical intent reveals it; a separately tested runtime policy may detect some instances. It is an opportunity for a high-leverage intervention because a cluster of questions sits near the same ranking-vs-breakdown distinction. A prompt clarification or correction rule can target that boundary, but its net effect and regressions must be measured on the complete configuration rather than inferred from the diagnosis.

Related chapters

• Trial-level traces — reading the failing cases
• Confusion analysis — the directional ranking-for-breakdown swap
• Semantic boundary failures — why this is the hardest, most instructive kind
• Guardrail experiments — the fix this diagnosis made possible