The experiment

Cost and latency, honestly

Cost and latency are inputs to a routing decision — not clean model rankings.

70%80%90%100%$1$2$5$10$20cost per 1,000 questions (log)Qwen 3B+GR · $0.80Qwen 7B+GR · $1.66Claude Haiku 4.5 · $5.32Claude Sonnet 4.6 · $15.97OpenAI Small · $3.93OpenAI Strong · $19.64
Accuracy against marginal cost (log scale). The two systems tied for highest accuracy sit at very different costs — the decision is a price/quality trade, not a single winner.

Cost here is a marginal dedicated-GPU estimate (summed request time); it excludes startup, idle, and scale-down, and the hosted systems are token-priced — so it is not a like-for-like production bill.

Latency is deliberately not used to rank models: the legacy harness did not control per-request client reuse, so those numbers are directional only. The production takeaway is a routing policy — send most traffic to a strong-and-cheap system, escalate hard cases — not a single champion.