Measured worked evaluation
Which complete system should a team deploy?
A team needs to turn plain-English business questions into reliable structured query intents. Six complete system configurations answered the same 88tasks; the decision uses accuracy, failure behavior, cost and operational trade-offs.
1 · Understand the opportunity
Turn business questions into dependable structured intent.
Analysts and application users ask questions in ordinary language. The system must translate each question into the correct intent and fields for a downstream query layer—without silently confusing rankings, breakdowns, filters or aggregations.
- Desired outcome
- Correct structured intent that downstream software can execute safely.
- Success measure
- Exact field-for-field match against a known-correct answer.
- Constraints
- Schema-valid output, useful latency and cost, and failures that can be diagnosed.
- Out of scope
- This run does not measure generated SQL execution or production user behavior.
2 · Evaluate complete options
Compare the whole answering system—not only the model.
Each option includes its model, prompt, output schema, validation and any correction policy. All 6 options answered the same development, held-out and adversarial question sets, producing 528 inspectable trials.
3 · Choose the best fit
Default recommendation: Claude Haiku 4.5
This system is the strongest default candidate for the measured workload: it achieved joint-best observed overall accuracy at 95.5%, including 60 of 60 held-out tasks. It tied OpenAI Strong on accuracy while carrying the lower marginal estimate in this run ($5.32versus $19.64 per 1,000 sequential questions). That is a scoped recommendation; the cost estimates are not like-for-like production prices, and this is not a universal model ranking.
Accuracy-first deployment for this structured-intent task and schema.
Adversarial failures remain, and latency and marginal cost are directional rather than clean production rankings.
A different task distribution, SLO, price model, runtime or policy can make another complete system the better choice.
4 · Inspect the evidence
Trace the recommendation back to individual attempts.
The evidence view exposes each question, selected systems’ answers, verifier outcomes, policy interventions and the frozen run metadata behind the aggregate results.