Part 5 — Failure Analysis and System Improvement

Parsing, validation and execution failures

Where failures live·Evaluation·7 min read

Not every failure is a wrong answer. Some outputs never parse, fail the schema, or break when executed — structural failures that live at a different boundary than semantic ones and need a completely different fix. This chapter sorts failures by which boundary they hit, so you apply plumbing fixes to plumbing problems.

The last two chapters were about outputs that were valid but wrong. This one is about their opposite: outputs that never got the chance to be wrong because they were malformed — they didn't parse, failed the schema, or crashed on execution. These structural failures sit at a different boundary than semantic ones, they announce themselves loudly (unlike the silent boundary failures), and they demand a completely different class of fix. Sorting failures by which boundary they hit is what keeps you from applying a prompt tweak to a plumbing bug.

What you will understand by the end

  • The three structural failure stages: parsing, validation, and execution.
  • Why these are syntax/plumbing failures, categorically different from semantic ones.
  • Why they're easier to catch but can mask each other and semantic problems.
  • How to route each failure to the right kind of fix by which boundary it crossed.

Three ways an output fails before meaning

Recall the pipeline: after the model produces text, it must be parsed, validated, and executed. Each is a place a structural failure can happen:

  • Parsing failure. The raw output can't be turned into the expected structure at all — truncated JSON, extra prose around the object, a malformed field. The pipeline chokes before anything downstream runs.
  • Validation failure. It parses, but violates the schema — a missing required field, an out-of-range value, a wrong type, a disallowed enum. Structurally shaped but not schema-legal.
  • Execution failure. It parses and validates, but breaks when used — the query it builds errors, references a non-existent column, times out, or returns nothing.
   raw output ─▶ PARSE ─▶ VALIDATE ─▶ EXECUTE ─▶ (then: is it semantically right?)
                  │          │           │
              can't form   schema      breaks
              structure    illegal     when run
              └──────── structural failures: wrong SHAPE, not wrong MEANING ───────┘
Key idea

Structural failures are failures of shape and mechanics, not meaning — the output never became a well-formed, runnable thing, so the question "is it correct?" doesn't even arise yet. They belong to a different boundary than semantic failures, and they take different fixes: schema, prompt format, parser robustness, and plumbing — not teaching the model a distinction.

Loud failures, different fixes

Structural failures are, in one way, the easy kind: they're loud. A parse error throws, a validation check returns false, an execution errors — you know immediately that something broke and often exactly where. Contrast the silent semantic boundary failure, which sails through every check looking perfect. The fixes are correspondingly mechanical:

  • Parsing → constrained decoding to guarantee shape, a more robust parser, or stripping wrapper text.
  • Validation → tighten or correct the schema, or add prompt instructions and examples that keep the model in-bounds.
  • Execution → validate references and constraints before running, add fallbacks, handle timeouts and empties.

The one thing you must not do is mistake a structural failure for a semantic one (or vice versa) and aim the wrong fix at it — retraining the model to fix a truncation bug, or hardening the parser to fix a wrong-pattern error.

Watch out

Structural checks can mask semantic ones and each other. If most outputs fail to parse, you can't even see whether the ones that do parse mean the right thing — the structural problem hides the semantic signal. Fix the loud structural failures first to clear the view, then measure semantic correctness on the outputs that now make it through. Order matters: shape before meaning.

Sort every failure by its boundary

The organising move of this chapter is a first cut on every failure: which boundary did it cross? Did it fail to parse/validate/execute (structural) or did it run cleanly and mean the wrong thing (semantic)? That single split routes the failure to the right family of fix and keeps your error taxonomy honest — structural and semantic failures counted together would hide which kind of engineering the system actually needs.

Observed evidence

This project makes the split unusually clean: constrained decoding drove parse and validation failures to essentially zero — every one of the six systems produced 100% schema-valid JSON — so the structural boundary was solved and all the remaining failures were semantic (wrong-side-of-boundary intents). That's the ideal diagnostic state: with structure guaranteed, the 45.5%–95.5% spread is purely a meaning story, and you know exactly which kind of fix applies. 100% valid, all failures semantic →

Mental model

Before an output can be wrong, it has to be well-formed and runnable. Parsing, validation, and execution failures are structural — wrong shape or broken mechanics, not wrong meaning — and they take plumbing fixes (schema, parser, constrained decoding, guards), not model teaching. Sort every failure by which boundary it crossed, fix the loud structural ones first to clear the view, then measure meaning.

Common mistakes

  • Lumping structural and semantic failures together. They need different fixes; counting them as one hides which engineering the system needs.
  • Aiming the wrong fix. Retraining to fix a truncation bug, or hardening the parser to fix a wrong-pattern error — mismatched boundary and fix.
  • Measuring semantics through a structural fog. If lots of outputs don't parse, semantic accuracy on the rest is unmeasurable until you fix the shape.
  • Ignoring execution. An output that parses and validates can still break when run; execution is a real, separate failure stage.

Practical guidance

  • Classify each failure by boundary first — parse / validate / execute (structural) vs runs-but-wrong (semantic) — before choosing a fix.
  • Solve the structural boundary with constrained decoding, robust parsing, tighter schemas, and execution guards — mechanical fixes for mechanical failures.
  • Fix structural failures first to clear the view, then measure semantic correctness on what gets through.
  • Keep structural and semantic failure counts separate in your taxonomy and dashboards.

Summary

  • Parsing, validation, and execution failures are structural — wrong shape or broken mechanics, not wrong meaning.
  • They're loud (unlike silent semantic failures) and take plumbing fixes: constrained decoding, robust parsers, schemas, execution guards.
  • Structural failures can mask semantic signal; fix them first, then measure meaning.
  • Sorting every failure by which boundary it crossed routes it to the right fix — this project solved structure (100% valid) so its remaining failures were purely semantic.

Knowledge check

30% of your system's outputs fail to parse, and of the rest, semantic accuracy looks "okay." Why can't you trust that semantic number yet, and what's the fix order?

Because the 30% parse failures mask the semantic picture: you're only measuring meaning on the 70% that happened to parse, which may be a biased, easier subset — the malformed 30% could be concentrated on hard inputs whose semantics you're not seeing. Fix the structural boundary first (constrained decoding / robust parsing to get outputs well-formed), then re-measure semantic correctness on the now-complete set. Shape before meaning: you can't assess correctness on outputs that never became well-formed.

A batch of failures all produce valid, schema-legal JSON that selects the wrong metric. Is this a parsing/validation problem? What does that imply about the fix?

No — it's a semantic failure, not structural. The outputs parse and validate perfectly; they just mean the wrong thing (wrong-side-of-boundary). So plumbing fixes (better parser, tighter schema, constrained decoding) won't help — those address shape, and the shape is already correct. The fix belongs to the semantic family: prompt clarification, contrasting examples, or a correction rule for the distinction. Aiming a structural fix here would waste effort on a boundary that isn't the problem.

Related chapters