Part 10 — Integrated Case Studies

The structured-intent use case

Setting up the case study·Evaluation·6 min read

The final part stops teaching in the abstract and walks one real experiment end to end. It opens with the task the whole book has used as its example — turning a business question into a validated query intent — and the single design decision that made it cleanly evaluable.

Every concept in this book has been illustrated by the same running example — a natural-language-to-query system — and this closing part finally examines it as what it is: a complete, worked case study. One task, one dataset, six systems, a real result, and every idea from the preceding parts visible in practice. It begins where the project began: with the use case and the one architectural decision that shaped everything downstream — having the model produce a validated intent, not a query.

What you will understand by the end

  • The task the case study solves, and why it was chosen as the book's example.
  • The pipeline: question → validated QueryIntent → deterministic query.
  • The design decision — intent, not SQL — and why it makes the system evaluable.
  • How this grounds the rest of Part 10.

The task: a business question in, a correct query out

The system takes a plain-English business question — "Revenue by region last quarter", "Top 5 customers by revenue" — and produces the structured query that answers it. It's a genuine, common application shape, and it was chosen as the book's example because it combines several patterns at once (application patterns): it's classification-like (which query shape does this question want?), extraction-like (which metric, dimensions, filters?), and heavily structured-output. One task exercises per-class accuracy, field extraction, and the valid-vs-correct distinction together.

Key idea

The natural-language-to-query task is a good teaching vehicle because it packs several patterns' difficulties into one evaluable problem — a boundary decision (ranking vs breakdown), field extraction, and structured output. Everything the book taught about evaluation shows up here, on one concrete system.

The pipeline: intent, then query

The system's request path is the anatomy made specific:

   "Revenue by region last quarter"
     → prompt: schema + glossary + rules + the question
     → model: produces a QueryIntent (pattern, metric, dimensions, filters, sort, limit)
     → validate the intent against the schema
     → optional correction policy (fix a known intent mistake)
     → deterministic SQL builder turns the intent into a query
     → run the query

The model never writes SQL. It produces a structured QueryIntent — the subject of the next chapter — and deterministic code turns that intent into a correct query.

The decision that made it evaluable

That boundary — model produces intent, code builds the query — is the single most consequential design choice in the whole project, and it's a direct application of the book's recurring advice to keep deterministic work out of the model:

  • The model does only the language-shaped step (understand the question → a structured intent), which is the part it's actually good at.
  • Deterministic code does the mechanical step (intent → SQL), which it does reliably every time.
  • And crucially, the intent is a small, canonical object you can score by exact match — far easier to evaluate than free-form SQL, which has many valid spellings.
Key idea

Asking the model for a validated intent instead of raw SQL narrows what it must get right, makes wrong answers hard to represent, and — decisively — gives the task a single canonical answer per question, which is what lets the whole experiment be scored deterministically and reproducibly. The evaluability of the case study was designed in, at the architecture level.

Observed evidence

You can inspect this exact system as data: six complete configurations on the same questions, every trial openable to see the question, the produced intent, and the expected one. The rest of Part 10 walks how it was built, evaluated, and interpreted — but you can browse the artifact now. Open the case study's systems →

Mental model

The case study is a natural-language-to-query system: question → validated QueryIntent → deterministic query. Its defining decision is that the model produces an intent, not SQL — keeping the model to the language-shaped step and giving the task a single canonical answer that can be scored by exact match. That one choice is why the whole experiment is cleanly evaluable, and it grounds the rest of this part.

Common mistakes

  • Asking the model to emit SQL directly. Free-form SQL has many valid spellings (hard to score) and lets the model get deterministic details subtly wrong.
  • Treating this as a niche task. Its combination of classification, extraction, and structured output makes it a stand-in for a huge class of LLM applications.
  • Missing why it's evaluable. The single-canonical-intent design is what makes exact-match scoring possible; it wasn't luck.

Practical guidance

  • For your own task, find the analogous split: let the model do the language-shaped step and keep the deterministic work in code.
  • Design the model's output to have a canonical form where you can — it's what makes the task cheaply and reproducibly evaluable.
  • Read the rest of Part 10 as a worked example — schema, dataset, diagnosis, guardrails, comparison, interpretation — to see every earlier concept applied.

Summary

  • The case study is a natural-language-to-query system, chosen because it combines several application patterns in one evaluable task.
  • Its pipeline is question → validated QueryIntent → deterministic query; the model never writes SQL.
  • The decision to produce an intent, not SQL keeps the model to the language step and gives the task a single canonical answer — the root of its evaluability.
  • The rest of Part 10 walks the experiment built on this system, end to end.

Knowledge check

Why does having the model produce a structured intent, rather than SQL directly, make the whole case study easier to evaluate?

Because the intent has a single canonical form per question, so it can be scored by exact match against one expected answer — cheap, deterministic, and reproducible. Free-form SQL, by contrast, has many equally-correct spellings (different ordering, aliases, join styles), so there's no single reference to match and you'd need semantic or execution-based scoring with all its ambiguity. Producing an intent also narrows what the model must get right (the language-shaped understanding) and hands the mechanical SQL-building to deterministic code — so the model's output is both easier to get right and easier to grade.

Why is a natural-language-to-query task a good stand-in for LLM applications generally?

Because it combines the difficulties of several application patterns in one task: it's classification-like (choosing the query shape), extraction-like (pulling out the metric, dimensions, and filters), and structured-output-heavy. So it exercises per-class accuracy, field-level extraction, and the valid-vs-correct distinction all at once — the same challenges that show up across classification, extraction, RAG, and tool-use systems. Lessons learned evaluating it transfer broadly, which is exactly why the book used it as its running example.

Related chapters