Reference
A1.5 run manifest
One place that records exactly what was run: the identifiers, versions, and settings behind every number on this site, and the parts of the record that are incomplete.
Elsewhere the interface uses readable labels (“OpenAI Strong”) because they are easier to compare at a glance. The exact identifiers live here, so nothing is hidden — only kept out of the way.
Systems under test
| Label on this site | Provider | Model identifier | Correction policy |
|---|---|---|---|
| Qwen 3B + guardrail | baseten | Qwen/Qwen2.5-3B-Instruct | ranking_guardrail |
| Qwen 7B + guardrail | baseten | Qwen/Qwen2.5-7B-Instruct | ranking_guardrail |
| Claude Haiku 4.5 | bedrock | us.anthropic.claude-haiku-4-5-20251001-v1:0 | none |
| Claude Sonnet 4.6 | bedrock | us.anthropic.claude-sonnet-4-6 | none |
| OpenAI Small | openai | gpt-5.6-luna | none |
| OpenAI Strong | openai | gpt-5.6-sol | none |
Self-hosted systems ran on a dedicated Baseten deployment; the Claude systems ran on Amazon Bedrock; the GPT systems ran on the OpenAI API. Deployment handles and endpoint hosts are operational detail and are deliberately not published — the model identifier is what determines the result.
Versions
| Run | a15-model-comparison-2026-07-24 |
| Dataset | [email protected] — 88 tasks (20 development, 60 holdout, 8 adversarial) |
| Prompt | baseline-intent-prompt@1 |
| Scorer / verifier | exact-query-intent@1 — field-for-field equality on pattern_id, metric, dimensions, filters, sort, limit |
| Correction policy | ranking_guardrail, a deterministic post-model rule. Applied to the two Qwen systems; the other four ran uncorrected. |
| Execution date | Jul 24, 2026 |
Decoding and reasoning settings
All three provider paths request a strict JSON schema rather than free text, and cap the response at 256 output tokens. Sampling controls differ because the APIs differ — which is itself part of the configuration:
| Path | Settings sent |
|---|---|
| Bedrock (Claude) | temperature: 0, maxTokens: 256, json_schema output format |
| OpenAI (GPT) | reasoning_effort: "none", max_completion_tokens: 256, response_format: json_schema (strict). No temperature is sent. |
| Baseten (Qwen, vLLM) | max_tokens: 256, vLLM structured_outputs schema extension. No temperature is sent. |
Execution
One. No retries, no best-of-n, no self-consistency. Every score is a single sample.
88 tasks × 6 systems = 528 trials, run sequentially.
20 development · 60 holdout · 8 adversarial. Only the held-out split is an unbiased generalization test.
Artifacts
- a15.json — the complete public dataset: every system, metric, and individual trial, including each raw and corrected intent. This page and the whole site are computed from it.
- Per-system aggregates and the full trial list are in that same file; there is no separate summary artifact to fall out of sync with it.
Known limitations
- Single attempt. No run-to-run error bars on any number here. Repeatability was measured only on a different, much smaller workload — .
- 88 tasks. Small enough that a few tasks move a percentage point. Differences of a point or two are not resolvable at this sample size.
- The development split was tuned on. Scores there are descriptive, not generalization. The 8-task adversarial split is diagnostic only — far too small to rank systems.
- Latency is directional. The harness did not control per-request client reuse, so latency is not a fair cross-system ranking and is not used as one.
- Cost is a marginal estimate, not a bill: summed request time for the dedicated GPU, token pricing for the hosted systems, excluding startup, idle, and scale-down.
- Synthetic data. The questions and the database are constructed for this evaluation. No user or customer data is involved.
Gaps in this record
- Cost derivation is not reproducible from the published artifact. The per-1,000-question figures were computed outside the run script, against a price list that was not captured with the run — so there is no pricing date to cite. Treat the cost column as an estimate of the right order, not an auditable number.
- The exact code revision is not pinned. The archived run files carry a different metric set than the current evaluation script emits, so the script changed after the run and the run was not tagged. The measurements are unaffected — the trials are recorded per-task — but re-running today would not reproduce the artifact byte-for-byte.