Reward Latency: The Instrument Broke More Than the Subject

August 22, 2026

Wanted to know what a code-execution reward costs a GRPO loop. Ended up spending most of the study debugging my own measuring equipment. Write-up: Price the Reward Function.

Aug 21: The premise died first

The plan was a post about synchronous RLVR starving GPUs, with a 70% idle figure. A research pass killed it before any code: AReaL published reward/generation overlap in May 2025 and named unit-test execution specifically, and RollPacker has measured the reward stage at 5 to 13% of step time on a real code sandbox.

So the architecture was never the contribution. Reframed around measuring the barrier instead, which turned out to be the better post anyway.

Aug 21: NVML does not work in a container

Day-zero gate, one cent. The plan was to read GPU occupancy from NVML’s buffered sample API. Inside a Modal container it returns “Not Supported” on both a T4 and an L4.

The fallback, the number nvidia-smi reports, does return values. I drove a square wave with a duty cycle I measured at 49.7% and asked the API to recover it seven times per GPU. T4 came back at 83.7% mean, L4 at 56.0%, both swinging about 22 points trial to trial against a stimulus whose own interval is four ten-thousandths wide.

First version of that probe was also wrong: it queued matmuls without synchronising, so the GPU drained the queue through the sleep and the wave was never square. It read ~100% busy and I nearly published that as evidence NVML misreports. An instrument that does not measure its own stimulus cannot tell a broken instrument from a broken subject.

Torch profiler kernel intervals recover the wave to within 4 to 9 points, biased low and precise, which is the right shape for an instrument. That became the metric.

Aug 21: The ceiling was the wrong shape

Budgeted ±$360 on the question of whether sandboxes count against Modal’s 100-container Starter limit. They do not. The legacy backend rate-limits creation at 5/s, and MODAL_SANDBOX_V2=1 clears it: 256 of 256 concurrent creates succeeded, p50 363 ms.

Settled for ten cents. No plan upgrade.

Aug 22: Everything was single-burst

Asked whether the day-zero numbers were statistically sound. They were not. Every cell was one burst. Large samples inside one moment, which is easy to mistake for repetition, and zero run-to-run repeats anywhere.

Re-ran at seven repeats. η, the warm-pool ceiling, moved 0.317 → 0.372. The NVML claim turned out to be too strong: on the L4 the bias is not statistically established at n=7, only the swing is. Weakened the claim to precision rather than accuracy.

Also found I had quoted a trial I had already invalidated. Fixed the broken-wave bug, never propagated the exclusion into the numbers derived from it.

Aug 22: T_train equals T_gen

Gate A. Qwen3-0.6B on one L4: generation 12.47 s, training 12.20 s. Ratio 0.978. The τ ladder had been derived from a table assuming training was free, so it was wrong by a factor of two in the denominator.

It OOM’d first, and that was lucky. log_softmax(logits.float()) over Qwen3’s 151,936-token vocabulary materialises 2.9 GiB per micro-batch of 4. Real trainers use a fused selective log-softmax. Had the card been slightly bigger it would have returned a plausible, too-large T_train, and the whole ladder would have been built on it.

Aug 22: The finding

Held the median reward time at exactly 2 seconds and varied only the fraction of completions hitting the 30 s timeout.

Same median. 7.5% or 55.0%. Nothing in between, because the barrier is a max: GRPO waits on the slowest completion in the batch. At a 1% per-completion rate with 128 completions, 72% of steps contain at least one.

It saturates. 2%, 5% and 10% all read 55.0%, because once at least one timeout per batch is near certain, more of them change nothing.

Aug 22: A recommendation that was exactly backwards

Wrote up “reduce completions per barrier” as the second-best fix. A hostile review pass caught it. Verified against my own two published constants:

Busy time shrinks with the batch; the timeout constant does not. Smaller barriers lower the odds of eating the timeout and raise what it costs, nearly fourfold per completion. The real fix is finer-grained barriers without shrinking the batch, since GRPO normalizes over a group of 8 rather than the whole batch.

Same review pass caught three more: “cutting the rate in half buys nothing” is only true above the knee, “the mean is useless here” was wrong because wall clock over a run is a sum over steps, and my 55% headline is the same identity I disown for the curve, read at x=30.

Where it landed

Seventeen predictions registered before the runs, each with a stated falsifier. Nine have verdicts, three of them refuted, including one about my own instrument. Pre-registering was never about avoiding being wrong. It was about making being wrong visible.

Seventeen bugs. Thirteen of them in my own tooling rather than in the thing being measured: a probe that misread its own stimulus, a verdict line comparing the wrong quantity, an attribution-checking git hook that blocked its own definition, and code requiring Python 3.12 in a package declaring support for 3.11.

Whole study came to about $7 of compute, roughly two thirds of which bought an idle GPU. That is not an accident of the method. It is the measurement.