Reference

Glossary

System

A complete configuration: model + prompt + output schema + correction policy. The evaluated object here is the system, not the bare model.

Task

One question paired with its expected structured query and the split it belongs to.

Trial

One system’s single attempt at one task. This run uses one attempt per task, so tasks × systems = 528 trials.

QueryIntent

The structured object a system must produce (metric, dimensions, filters, sort, limit…). Deterministic code turns it into SQL; the model never writes SQL directly.

Exact match

Scoring rule: a trial is correct only if its intent equals the expected intent field-for-field. Superficially reasonable but off-by-one-field answers count as wrong.

Development split

The tuning questions (informed the prompt/guardrail). Scores here are descriptive, not unbiased generalization.

Held-out split

Questions no system was tuned on — the main generalization test.

Adversarial / stress set

A small set of deliberately misleading questions. Diagnostic only; too few to rank models.

Correction policy (guardrail)

A deterministic offline rule applied after the model to fix a known failure pattern. Can fix some cases and regress others.

Raw vs corrected

Raw = the model’s own validated output. Corrected = after the correction policy. Reporting both separates model quality from policy quality.

Intervention · fix · regression

An intervention is a case the policy changed. A fix turns wrong→right; a regression turns right→wrong.

Schema compliance

Whether the output is valid against the required JSON schema. Every system here hit 100% — valid JSON is not the same as a correct answer.

p50 latency

Median response time. On this site it is directional only (the legacy harness did not control client reuse).

Marginal cost / 1K

Estimated cost per 1,000 sequential questions. For dedicated GPU it is summed request time and excludes startup/idle/scale-down; hosted systems are token-priced.