45.0% and 44.0%

September 15, 2026

Originally published at Poisson Labs.

I ran the same agent benchmark twice. Same configuration, same tasks, same seeds, nothing changed in between.

The first run scored 45.0%. The second scored 44.0%. A point apart, which on any leaderboard is the same number twice.

Then I compared the two runs task by task instead of score to score.

Forty-four of the 200 matched tasks came out differently: solved in one run, unsolved in the other. Every one of the 200 pairs, including the 156 that agreed on the final answer, took a different path to get there. Not one trajectory was reproduced.

This is the short version. The full post, with the 2×2, the per-call hazard, both readings I withdrew and the provenance of every number, is on Poisson Labs.

Two things are worth keeping separate. Outcome disagreement is the 44, meaning pairs where the verdict flipped. Trajectory divergence is the 200, meaning pairs where the sequence of model calls differed whether or not the verdict changed. The first is a fifth of the tasks; the second is all of them.

The flips cancelled, which is why the summary looked calm. Twenty-three tasks went solved to unsolved and twenty-one went the other way, exact McNemar p = 0.880. That symmetry is a fact about this pair of runs and not a rule. With 23 and 21 out of 200 the interval on the difference runs to roughly ±6 points, so another pair of arms could land several points apart on noise alone.

Divergence happens throughout the run

My first guess was drift. Agent trajectories are long, small differences compound, so of course two runs end up apart eventually.

That is not what the data shows. I measured the conditional hazard, meaning the fraction of pairs still identical entering a given model call that first diverge at that call. Over the first three calls it runs 0.470, 0.594, 0.488, with overlapping intervals. No increase, so divergence is not something that builds toward the end of a trajectory. No decrease either, so it is not concentrated at the start.

I am not claiming the calls are independent, and I did not test it. A flat-looking hazard over the calls where I have support is not the same as a constant process. What it does rule out is the drift story: nothing here suggests divergence accumulates with trajectory length.

All 200 first divergences look the same. The request was byte-identical and the response differed, so the first observable difference sits at the model-response boundary. None entered first on a tool response, none on ordering.

Matching seeds did not help here

I ran the comparison again with different seeds instead of matched ones. Same seed: 22.0% of pairs disagree. Different seed: 23.3%. The intervals overlap almost entirely.

Matching the seed did not measurably reduce disagreement in this comparison, and that is the whole claim. It is not an argument that seeds are useless; they pin what they pin, and the provider documents the parameter as best-effort. It is an observation about this substrate and this provider.

Two things I got wrong

Before the run I declared that a difference in the provider’s backend fingerprint would name a cause for the divergence. I withdrew that. At the first span the request is identical by construction, so a fingerprint difference there is whatever rate the provider happens to route differently. That base rate is 49.5%, against 56.5% at the first divergence, Fisher p = 0.193. The difference is not distinguishable from the base rate, so it does not support the reading I had declared. I am not concluding the reverse either, because a non-significant result is not evidence of no association. I only caught it because I had built the base-rate control.

My first span comparison was byte-for-byte, which scored two episodes making the identical tool call as divergent, because the provider mints a fresh identifier for every tool call and echoes it into the next request. Normalising that identifier and nothing else dropped the apparent hazard at call 3 from 1.00 to 0.14. Nothing in the headline moved: all 200 pairs still diverge, 94 of 200 still diverge at call 0, and the 44 flips come from stored outcomes rather than from spans.

Both withdrawals are in the full post. A post arguing that published numbers do not carry their uncertainty should show the readings its own author had to retract.

The receipt

τ²-bench airline, 50 tasks, 4 replicates per task, two arms run back to back inside a twenty-minute window. 400 episodes, 200 matched pairs, $5.45 all in at about $0.0136 an episode. Within each pair the config hash and the seed are identical by construction, and I asserted that per pair rather than assuming it.

Outcome disagreement was 44/200, or 22.0%, with a task-clustered 95% interval of 0.155 to 0.285. That is a rate for this benchmark, this model pin, this provider and this twenty minutes. It is not a constant, and I do not know the rate over longer windows. What I would expect to travel is the shape rather than the magnitude: a stable aggregate can sit on top of unstable task outcomes, matching seeds may not stop it, divergence is not a function of trajectory length, and it varies by task.

The measurement ran through Proofload, the statistical instrument I build at Poisson Labs. Every number above traces to a committed analysis over the two stored runs, and the full post is at poissonlabs.ai/research/the-same-run-twice.