Reference

Companion experiment: run-to-run variance

A separate, smaller experiment that tests one assumption the main comparison relies on: that running the same inputs again returns the same answers.

This is not the A1.5 comparison.Different model, different deployment, different dataset, different question. The six-system comparison elsewhere on this site evaluates hosted and self-hosted configurations on 88 tasks; this page reports a 20-question repeatability check on a single self-hosted deployment. Its numbers are not comparable to the leaderboard’s and are not part of that run.
System under test

Qwen 2.5 3B, vLLM 0.12.0, one dedicated Baseten deployment on a single NVIDIA L4.

Workload

20 gold questions, replayed at concurrency 1, 16 and 32. Temperature 0.

Question asked

Does batching under load change what the model produces?

Concurrency is the variable that matters here: it is what changes how requests are batched together, and batch composition is the most commonly cited reason that identical requests can return different text at temperature 0. If batching were going to perturb outputs, a concurrency sweep is where it would show.

Result: it did not. At every level, 15 of 20 raw intents matched the expected answer, and zero intents differed from the sequential (concurrency-1) baseline — the outputs were bit-identical across the sweep. The companion throughput measurement over the same session was likewise highly reproducible: across three repeated 150-request sweeps, throughput varied by under 1% at each level.

What this does and does not establish. It is direct evidence that batching did not flip an output for this model, this deployment, this 20-question set, and this range of load. It is notevidence that temperature-0 inference is deterministic in general, that batched kernels are always bit-identical, or that a different model, runtime, GPU, or larger dataset would behave the same way. Twenty questions is a small sample, and a result of “no differences observed” bounds how often differences occur — it does not show the rate is zero.

The wider point is the reason this page exists at all: repeatability is something you measure and scope, not something a setting guarantees. Tokens, context and generation works through why that is, and what to measure instead.