Part 10 — Integrated Case Studies

Diagnosing intent failures

Diagnosis and correction·Evaluation·6 min read

Part 6's failure-analysis toolkit, run on the real experiment. This chapter reads the case study's failing trials, groups them into a taxonomy, finds the one dominant confusion, and names it as a semantic boundary failure — the diagnosis that drove everything the project did next.

With a schema, a dataset, and systems producing outputs, the case study turned to the question Part 6 is about: why do the failures happen? This chapter is failure analysis run on the real experiment — opening the failing trials, sorting them into types, drawing the confusion, and localising the dominant failure to a single semantic boundary. That diagnosis is not an academic exercise: it's what told the project exactly what to fix, and it's why the guardrail that came next could be so targeted.

What you will understand by the end

Reading the failing trials

The diagnosis started where Part 6 says it must: not with the aggregate, but with the individual failing trials. Each trace shows the question, the raw model intent, and the expected intent side by side, so a failure is legible — you can see which field diverged and how. Reading a batch of them, a pattern jumps out immediately, one that no aggregate accuracy could have revealed: the failures are not scattered, they cluster on one kind of mistake.

The taxonomy and its dominant type

Sorted into an error taxonomy, the failures fall into a small set of types — wrong pattern, wrong filter, wrong metric — and the distribution is lopsided: the dominant type by far is wrong pattern, and within it, one specific swap. Ranking by frequency turns the pile of failures into a clear priority: fix the wrong-pattern class and you fix most of the gap.

Key idea

The case study's failures aren't diffuse weakness — they concentrate in one type. Typing and counting them (Part 6's discipline) converted "the 3B system scores 45.5%" into "it gets the query shape wrong on a specific class of questions," which is an actionable, targetable statement rather than a number to despair at.

The confusion: ranking for breakdown

Drawing the confusion sharpens it further. The dominant off-diagonal cell is directional: the model predicts top_n_by_metric (a ranking — top-N by some metric, descending, truncated) when the expected answer is metric_by_dimension_with_filter (a breakdown — group by the dimension, list all of them). The canonical instance is "Revenue by region last quarter":

   question:  "Revenue by region last quarter"
   expected:  metric_by_dimension_with_filter   (breakdown — all regions)
   model got: top_n_by_metric                    (ranking — top-10 regions, desc, limit 10)
              ▲ valid JSON, wrong query shape — off by exactly one semantic step

The model heard "by region" as a request to rank regions rather than break down revenue across them. Same words, adjacent meaning, opposite query.

A semantic boundary failure

This is a textbook semantic boundary failure: two correct answers sit a hair apart in meaning, separated by a subtle cue ("top" vs plain "by"), and the model lands on the wrong side. Crucially, it's valid but wrong — the intent is perfectly schema-legal, so no structural check catches it; only comparison against the expected meaning does. Naming the failure this precisely is what made the fix possible: a boundary failure points at the exact distinction to teach, and a whole cluster of questions shares it, so fixing the distinction fixes many cases at once.

Observed evidence

This one confusion explains the bulk of the weaker systems' gap: because it's concentrated and directional (ranking-for-breakdown, not random error), it was both diagnosable from the traces and fixable by a targeted rule — which is exactly why the correction policy that followed could add ~25 points on the 3B model. Diagnosis drove the fix. Open the failing trials and see the confusion →

Mental model

Diagnosing the case study is Part 6 in practice: read the failing trials (not the aggregate), sort them into a taxonomy (dominant type: wrong pattern), draw the confusion (directional: ranking-for-breakdown), and name it a semantic boundary failure (valid but wrong, "by region" misread). The concentration and direction are what made it both diagnosable and targetably fixable.

Common mistakes

  • Diagnosing from the aggregate. "45.5%" hides that the failures cluster on one shape mistake; you must read the trials.
  • Missing the direction. "Ranking and breakdown get confused" is weaker than "ranking predicted for breakdown" — direction names the fix.
  • Treating it as general weakness. The failure is one concentrated, learnable distinction, not diffuse poor performance.
  • Forgetting it's valid-but-wrong. No structural check catches it; only measuring meaning against the expected intent does.

Practical guidance

  • Start diagnosis by reading failing trials with expected beside actual, then type and count them to find the dominant class.
  • Draw the confusion to get the direction — it names the specific, fixable distinction.
  • Recognise boundary failures (valid but wrong, adjacent meanings) as high-leverage: one fix clears a cluster.
  • Let the diagnosis drive the fix — a precisely-named failure is what makes a targeted guardrail possible.

Summary

  • The case study's failures were diagnosed by reading trials, typing them, and drawing the confusion — Part 6's toolkit on real data.
  • The dominant failure is wrong pattern, specifically top_n_by_metric predicted where metric_by_dimension_with_filter was expected — ranking for breakdown.
  • It's a semantic boundary failure: valid but wrong, "by region" misheard as a ranking cue.
  • Its concentration and direction made it diagnosable and targetably fixable — driving the guardrail that followed.

Knowledge check

The 3B system scores 45.5%. Why is reading the failing trials more useful here than trying to improve the aggregate directly, and what did it reveal?

Because "45.5%" tells you the size of the problem, not its shape — you can't fix a number, only the specific failures behind it. Reading the failing trials (with expected beside actual) revealed that the failures concentrate on one type: the model chooses a ranking pattern (top_n_by_metric) where the question wanted a breakdown (metric_by_dimension_with_filter), misreading "by region" as a top-N cue. That turns a hopeless-looking number into a single, learnable, directional distinction to fix — which a targeted correction rule could then address for ~25 points, something no amount of staring at the aggregate would have produced.

Why does the ranking-for-breakdown failure evade the schema's validity check, and what makes fixing it high-leverage?

It evades validity because the wrong intent is fully schema-legal — top_n_by_metric is a valid pattern with valid fields, so it parses and validates perfectly; it's just the wrong pattern for the question. Only comparing against the expected meaning (the canonical intent) catches it. It's high-leverage to fix because it's a semantic boundary failure: a whole cluster of questions sits near the same ranking-vs-breakdown distinction, so a single fix that teaches that distinction (a prompt clarification or a correction rule keyed to it) resolves the entire cluster at once, not just one trial.

Related chapters