Structured Outputs Gave Me Valid JSON - and Semantically Wrong Answers

Part 2: What happened when I compared a 3B model, a 7B model, and four hosted APIs on the same NLQ-to-SQL contract

Every evaluated model achieved 100% schema compliance.

Raw semantic accuracy still ranged from 45.45% to 95.45%.

That gap became the next bottleneck in my NLQ-to-SQL system.

In Part 1, I fixed an upstream concurrency-propagation problem and increased inference throughput by approximately 22x. The endpoint could finally batch requests, prefix caching was effective, and the useful operating range of a single replica was measurable.

But serving optimization only made the answer arrive faster. It did not make the answer correct.

The next question was whether semantic quality should be improved with a larger open model, a deterministic guardrail, or a hosted API.

I evaluated six model endpoints on 88 natural-language questions. The existing ranking guardrail was also applied offline to the raw outputs of both Qwen models, avoiding duplicate inference calls.

The result was not a universal model winner. It was a narrower and more actionable decision:

Use Claude Haiku 4.5 as the provisional default for the next evaluation stage, keep Qwen 2.5 7B as the open-model baseline, and do not transfer the existing 3B guardrail to 7B unchanged.

Haiku tied the highest aggregate accuracy at 95.45%, answered all 60 holdout questions correctly, and had a more favorable cost profile than Sonnet or GPT-5.6 Sol in this workload.

The experiment did not prove that Haiku was more accurate than the other hosted models. The best and worst hosted results differed by only two questions.

That distinction matters. This was enough evidence to choose the next experiment, not enough evidence to declare a permanent production winner.

The Application Contract Stayed Deterministic

The application architecture remained:

Natural-language question -> constrained intent JSON -> deterministic SQL builder -> DuckDB

The model did not generate SQL directly. It generated a validated QueryIntent with six scored fields:

  • pattern_id
  • metric
  • dimensions
  • filters
  • sort
  • limit

A response failed exact-match scoring if any of these fields differed from the reference intent. For example, selecting the correct metric and dimension but adding an unsupported descending sort still counted as incorrect. The standard was stricter than merely producing executable or superficially reasonable SQL.

The same deterministic code then converted that intent into SQL.

This boundary separated three questions that are often collapsed into one:

  1. Did the provider return a response?
  2. Did the response satisfy the schema?
  3. Did the response represent the user's intended query?

All six model endpoints achieved 100% schema compliance in the measured run. Their raw semantic accuracy ranged from 45.45% to 95.45%.

Structured output therefore solved the syntax boundary, but it did not solve the meaning boundary.

The Evaluation Was Designed Around Generalization

The benchmark contained three groups:

Dataset Questions Purpose
Development 20 Preserve the original tuned reference set
Holdout 60 Test semantic generalization
Adversarial 8 Probe brittle ranking and filter language

The development set had already informed the 3B prompt and ranking guardrail. Its corrected score could not be treated as unbiased evidence.

The 60-question holdout was therefore the main generalization set. It contained 40 contrast-pair questions and 20 more naturalistic questions, with deliberate pressure on the boundary between:

  • "top customers by revenue," which is a ranking;
  • "revenue by customer," which is a grouped breakdown.

The eight adversarial questions were diagnostic. With only eight examples, one question moves the result by 12.5 percentage points, so that subset can reveal a failure mode but cannot establish a stable ranking.

Each model received the same logical intent contract. OpenAI, Baseten, and Bedrock used their respective structured-output interfaces. Bedrock also received an explicit schema instruction because its accepted schema subset differed from the application schema. The final Pydantic validation and exact-match scoring were the same.

This was a controlled provider-native comparison, not literal prompt identity.

The Aggregate Result

System Raw accuracy After current guardrail Marginal cost per 1K sequential questions
Qwen 2.5 3B / Baseten L4 45.45% 70.45% $0.80
Qwen 2.5 7B / Baseten L4 84.09% 84.09% $1.66
Claude Haiku 4.5 / Bedrock 95.45% n/a $5.32
Claude Sonnet 4.6 / Bedrock 94.32% n/a $15.97
GPT-5.6 Luna / OpenAI 93.18% n/a $3.93
GPT-5.6 Sol / OpenAI 95.45% n/a $19.64

The larger Qwen model produced the clearest open-model gain. Moving from 3B to 7B improved raw accuracy by 38.64 percentage points.

That was material, but 84.09% remained below the 90% gate chosen for this stage.

All four hosted systems cleared 90%, but narrowly:

  • Luna answered 82 of 88 questions correctly.
  • Sonnet answered 83.
  • Haiku and Sol answered 84.

The difference between the best and worst hosted result was only two questions.

The dataset breakdown was more informative:

System Development Holdout Adversarial
Qwen 3B raw 15/20 24/60 1/8
Qwen 3B + guardrail 20/20 41/60 1/8
Qwen 7B raw 20/20 51/60 3/8
Qwen 7B + guardrail 20/20 52/60 2/8
Claude Haiku 4.5 20/20 60/60 4/8
Claude Sonnet 4.6 20/20 59/60 4/8
GPT-5.6 Luna 20/20 57/60 5/8
GPT-5.6 Sol 20/20 59/60 5/8

Haiku's 60/60 holdout result made it the most useful provisional candidate. Its four failures were all adversarial filter mismatches, which also localized the next evaluation work.

But none of the hosted systems solved the adversarial set. Better model capacity reduced ordinary semantic errors without removing the need for an explicit policy for ambiguous or hostile phrasing.

Why the Aggregate Ranking Is Not the Decision

An 88-question benchmark can make small differences look more decisive than they are. The hosted systems differed by only two questions, and their Wilson 95% confidence intervals substantially overlapped.

That overlap is context, not a verdict: interval overlap alone does not test whether paired systems differ, and treating it as a test would only ever produce ties. Because every model answered the same questions, paired disagreement was the appropriate analysis rather than comparing percentages independently.

A two-sided exact McNemar test supported Haiku over Qwen 7B on this set: Haiku alone was correct on 10 questions, Qwen 7B alone was correct on none, and the exact p-value was 0.002.

The same analysis did not establish an accuracy difference between Haiku and the other hosted models:

Comparison Haiku only correct Other only correct Two-sided exact p
Haiku vs. Sonnet 1 0 1.000
Haiku vs. Luna 4 2 0.688
Haiku vs. Sol 2 2 1.000

The decision was therefore not "Haiku is proven best."

It was:

Haiku offered the strongest observed combination of holdout accuracy, aggregate accuracy, concentrated failures, and API cost for the next evaluation stage.

The Guardrail Did Not Transfer with the Model

The original 3B system had a narrow deterministic rule for a recurring error. When the model interpreted a plain grouped breakdown as a ranking, the guardrail could downgrade the intent from top_n_by_metric to metric_by_dimension_with_filter.

On the tuned development set, that rule looked excellent.

Across all 88 questions, its behavior was less clean:

Model Interventions Exact fixes Regressions Mixed Full-fix precision
Qwen 3B 28 23 3 2 82.1%
Qwen 7B 4 1 3 0 25.0%

For Qwen 3B, the guardrail still created a large net gain. For Qwen 7B, three of four interventions were regressions, and adversarial accuracy fell from 3/8 to 2/8.

One adversarial question regressed under the rule for both models.

This exposed an important systems lesson:

A guardrail is part of a model-specific error policy, not a universally safe postprocessor.

The 7B model made fewer false-ranking errors, so the same lexical trigger had a different precision profile. The rule's historical success on 3B was not evidence that it should sit unchanged in front of every future model.

Guardrails need the same evaluation discipline as models:

  • define the eligible error class;
  • measure intervention precision;
  • distinguish exact fixes from partial fixes;
  • count regressions explicitly;
  • rerun the audit whenever the underlying model or prompt changes.

Otherwise a visible rule layer quietly becomes a second, poorly measured model.

Cost Was a Routing Signal, Not a Universal Comparison

The hosted API estimates used observed input and output tokens with standard on-demand rates. The Baseten estimates multiplied summed request time by the listed L4 per-minute price.

Those are different economic measures.

Token-priced APIs charge for request volume. A dedicated deployment charges for compute while it is running, including deployment, scale-up, scale-down, and idle windows permitted by the autoscaling policy.

The Baseten figures in the table are therefore steady-state marginal estimates. They exclude image build time, weight loading, cold starts, a configured 15-minute scale-down delay, and billing-rounding effects. Actual experimental charges were higher.

The useful cost conclusion is narrower:

  • Luna was the least expensive hosted API in this measured workload.
  • Haiku cost more than Luna but substantially less than Sonnet or Sol.
  • Qwen on an L4 had the lowest marginal inference-window estimate, but its production economics depend on utilization, batching, replica uptime, and scale-to-zero behavior.

Cost per successful query, not cost per request, is the better production metric. That calculation also needs the real traffic shape and the cost of retries, escalation, and incorrect answers.

Why I Did Not Use Latency to Choose the Winner

The original benchmark recorded latency, but the legacy evaluator instantiated a new SDK client for every request, so connection reuse was not controlled consistently across providers.

The evaluator has since been corrected to reuse provider clients and to use an explicit inclusive percentile calculation. I did not rerun the paid benchmark only to repair the latency comparison.

The original latency values remain useful as directional run diagnostics, but they are not clean enough to rank the models.

This is also why the provisional decision rests on semantic accuracy, holdout behavior, failure concentration, and marginal cost - not on the lowest observed p95.

The Production Decision Is a Routing Policy

The evidence supports a staged architecture rather than a single-model claim:

  1. Use Claude Haiku 4.5 as the provisional default for the next evaluation.
  2. Keep Qwen 7B as the open-model optimization target.
  3. Remove or redesign the current guardrail for 7B.
  4. Reserve Sonnet or Sol as possible escalation routes only if a larger test demonstrates a quality gain that justifies their cost.
  5. Add clarification or rejection behavior for ambiguous requests instead of forcing every question into the nearest supported intent.

Before production selection, the next benchmark should freeze a larger production-like set covering ranking language, filter presence and absence, and the failure patterns where the hosted models disagreed.

It should then repeat:

  • every previous failure;
  • every cross-model disagreement;
  • a random sample of previous successes.

Three runs per selected item would begin to separate stable behavior from single-shot luck.

The success criterion should also move beyond aggregate exact match. A production decision needs:

  • request success;
  • schema compliance;
  • semantic accuracy among valid responses;
  • end-to-end semantic success;
  • regression rate for every correction layer;
  • latency from a corrected harness;
  • cost per successful query.

The Larger Lesson

Part 1 showed that continuous batching is an end-to-end serving property.

Part 2 showed that semantic reliability is also an end-to-end property.

A provider can guarantee valid JSON without guaranteeing the correct business meaning. A deterministic guardrail can fix a known error without transferring safely to a different model. A larger model can improve average accuracy without resolving adversarial ambiguity. A cheaper request can become expensive when it fails.

The model is one component in the decision path.

The real system is:

evaluation set -> provider response -> schema validation -> semantic policy -> deterministic execution -> operational routing

The next optimization is not simply "use the biggest model."

It is to make each boundary measurable, choose the cheapest system that clears the required quality bar, and route uncertainty deliberately instead of hiding it behind valid JSON.

Sources Checked

Official provider pages checked on 2026-07-24 support only the model names, capabilities, availability, and list prices cited in this article:

The local benchmark artifacts - not the vendor pages - support the reported accuracy, cost calculations, confidence intervals, paired tests, and guardrail outcomes. Raw provider payloads are not publication attachments because they contain diagnostic metadata unnecessary for reproducing the aggregate analysis.