Which complete system should a team deploy?

Every headline number is computed from deterministic, trial-level evidence: how tasks are scored by exact match, and why nothing on this site is hand-entered.

Development, held-out and adversarial tasks each answer a different question about a system — and only the held-out split is an unbiased test of generalization.

How to read the comparison as three separate questions rather than one score, and what the ranking does and does not tell you about a production choice.

The adversarial set shows every system failing cases a leaderboard position hides. Winning a benchmark is not the same as solving the task.

The same deterministic guardrail lifts Qwen 3B by 25 points and nets to zero on Qwen 7B. Why a single final score would hide that entirely.

Cost is a marginal estimate and latency is directional only. Both are inputs to a routing decision rather than clean rankings of model quality.

Continuous batching is an end-to-end property: a measured serving write-up on where throughput comes from, where the knee is, and what the queue does past it.

Constrained decoding drove structural failures to zero while semantic failures remained. Valid JSON is not a correct answer.

Which vLLM serving knobs changed measured throughput and tail latency on a single L4, isolated one flag at a time, and which made no difference at all.

The exact model identifiers, prompt and scorer versions, decoding settings, splits and limitations behind the A1.5 comparison — including the parts of the record that are incomplete.

A separate 20-question repeatability check on one self-hosted deployment: raw intents were bit-identical across concurrency 1, 16 and 32. Not part of the A1.5 comparison.