Price the Reward Function

August 22, 2026

A verifiable reward sounds like a function. In practice it is a container, a network round trip, and a process that might hang for thirty seconds.

That distinction does not matter when the reward is a regular expression checking whether the model wrote 42. It matters when the reward is a test suite, because then every candidate solution needs somewhere isolated to run.

In synchronous Group Relative Policy Optimization (GRPO), that execution becomes part of the training step. The step cannot complete until every reward evaluation completes or times out, which makes reward latency a scheduling variable rather than a property of the evaluator alone.

The experiment instruments a GRPO loop that scores generated code in a Modal Sandbox, holds generation fixed, and measures the effect on the step.

At a glance

• at an identical 2 second median reward time, GPU idle was 7.5% or 55.0%, depending only on whether one completion out of 128 hit the timeout
• at batch size 128, reducing the timeout rate from 10% to 2% barely changes expected step time, while 2% to 0.5% takes a quarter off the step
• warm-pooling the sandboxes, the fix I expected to reach for, turned out not to help
• three of the seventeen pre-registered predictions were refuted, including one about my own instrument
• the whole study cost about $7

The loop everyone copies

Modal ships a GRPO example that trains a model to solve coding problems. Its reward function creates a fresh Sandbox per completion, executes the generated code, reads the exit code, and terminates the Sandbox in a finally block.

# The shape from modal-labs/modal-examples, grpo_trl.py.
# One sandbox per completion, created and destroyed inline.
def compute_reward(code: str, test: str) -> float:
    sb = modal.Sandbox.create(app=app, image=image)
    try:
        proc = sb.exec("python", "-c", code + "\n" + test, timeout=30)
        proc.wait()
        return 1.0 if proc.returncode == 0 else 0.0
    finally:
        sb.terminate()

The example is doing what a teaching example should do. It caps at five steps and says in its own comments that the model is unlikely to learn well from this reward. The measurement targets what that shape costs at 128 completions.

The reason it costs anything is structural to GRPO. The algorithm samples a group of completions per prompt and normalizes the advantage within that group, so the trainer cannot compute a gradient until every completion has a reward.

The step waits on the slowest sandbox, not the average one.

Once the reward is part of a synchronous training step, the relevant quantity is its effect on the step.

Async pipelining is well-trodden ground (AReaL, DeepCoder, TRL). What has not been isolated is why synchronous setups degrade so sharply in the presence of tail latency. RollPacker reports 5 to 13% reward-stage idle on a much larger setup, while other posts have reported numbers closer to 70%. Both can be true. The difference comes from the interaction between tail latency and the synchronous barrier.

What the reward costs

A control comes first: hold the reward’s service time constant and confirm the harness measures what it claims. Qwen3-0.6B, a hand-rolled GRPO step, 16 prompts by 8 completions at 512 tokens, on a single L4. The reward’s service time is a controlled dial rather than a real test suite, so generation stays fixed while only the reward moves. 80 measured steps in randomised complete blocks.

GPU idle here means the fraction of step wall clock not covered by the union of GPU-busy phases, measured from instrumented spans and validated against CUDA kernel intervals from the profiler. I had planned to read it from NVML, the interface behind nvidia-smi, but inside a Modal container it could not recover a duty cycle I controlled.

Generation moved by 0.18% across an 80-fold change in reward time. Busy time, generation plus training, measured 24.51 seconds.

The curve validates the instrument. It is not the finding. It does not describe what a real reward distribution does to a synchronous barrier.

The finding is the barrier

Every rung above used a constant service time, so the slowest completion and the median completion were the same thing. Real rewards are not like that.

A 1% timeout rate does not mean 1% of steps are slow. With 128 independent chances to draw from the tail, it means the batch is very likely to be slow: the probability that at least one of 128 completions times out at a 1% rate is 72%. The median is the wrong statistic.

The second arm holds the median at exactly 2 seconds and varies only the fraction of completions that hit the 30-second timeout.

Same 2 second median. 7.5% idle in one case, 55% in the other.

The median didn’t change. The training step did. Nothing lands in between, and nothing can, because the barrier is a maximum: a step either contains a timeout and waits 30 seconds, or it does not and waits 2. Median reward latency does not determine the step time. The slowest completion does.

The quantity that decides your answer is the timeout divided by the busy time of the step it blocks. Mine is 30 over 24.5. RollPacker’s 5 to 13% on a 14B model across 32 H800s is consistent with the same arithmetic, because their busy time is far larger than mine. The statistic matters too, but scale alone explains much of the difference.

Median reward latency can therefore look healthy while the training loop still spends a large fraction of its time idle. They are measuring different parts of the same distribution. The relevant ratio is timeout divided by the busy time of the step it blocks.

The cost that isn’t on the graph

A timeout is usually scored zero, and GRPO normalizes rewards within the group. So an infrastructure failure gets laundered into a statement about the policy.

Take a group of 8 where every completion was actually correct. The right gradient is no gradient: all eight advantages are 0. Now one container crashes and is scored 0 instead:

Seven completions are rewarded for beating a container that crashed, and the eighth is pushed away from an answer that may have been right. A group with no signal in it has manufactured one.

In a more realistic group, half correct and half not, a correct completion that times out goes from an advantage of +1.00 to -0.77. The sign flips. The gradient does not lose a sample, it gets a wrong one.

This happens at a different rate from the stall, because the barrier is over the batch and normalization is over the group. At a 1% per-completion timeout rate, 72% of steps eat the full timeout while only 7.7% of groups carry a corrupted advantage. At 5%, essentially every step stalls and a third of all groups get a wrong gradient.

I did not measure what that does to training, and this study has no convergence run. The arithmetic above is the whole claim. The stall is the visible cost. This one does not show up on a utilization graph at all.

The order to fix it in

Idle is the same at a 2% timeout rate as at 10%, because once at least one completion is reliably slow the barrier is the timeout constant and further slow completions change nothing. That is true above the knee and false below it, and whether you are above it is a question about batch size:

At 128 completions per barrier, a 2%, 5% or 10% rate all put you past 92% certainty of paying the full timeout every step.

My timeout-to-busy-time ratio is 30 over 24.5, which is close to the worst case: a small model on one GPU makes the step cheap and the timeout expensive. Most real setups sit far below that, so expect the headline to shrink and the ordering to survive. The interventions, in order:

1. Lower the timeout constant. The barrier is the constant once the tail is reliably hit, so halving it halves the stall. RollPacker’s measured 1.6x from adaptive timeouts is this fix.

2. Push the rate below the knee, and do not bother shaving it above. Going from 5% to 2% takes my expected step from 54.5 seconds to 52.4, which is noise. Going to 1% takes it to 46.8, and to 0.5% takes it to 39.8.

3. Break the barrier without shrinking the batch. GRPO needs a group of 8 to normalize an advantage, not the whole batch, so a group’s backward pass can start as soon as that group is scored. Fifteen of my sixteen groups finish in about 2 seconds and then sit idle for 28 more waiting on one straggler. Running each group’s backward on arrival would hide 11.4 of the 12.2 seconds of training under that wait, taking a stalled step from 54.7 to 43.2 seconds. That is derived from the numbers above rather than measured, because I did not build it.

This is not asynchronous RL. Every completion still comes from the same policy snapshot and gradient accumulation is a sum, so you get the same gradient up to floating-point associativity. No staleness, no off-policy correction. It is scheduling.

Smaller batches look like the fix and are not. At N=8 a 5% rate hits only a third of steps, but busy time shrinks with the batch while the timeout does not, so each timeout costs nearly four times as much per completion. Finer-grained barriers, not smaller ones.

Warm-pooling appears nowhere on that list. I had expected it to be the first fix. It removes container boot, but boot is not what sets the barrier. Against a four-second suite even a perfect pool recovers about 11% of service time, and a pool wide enough for a batch of 128 costs $12.12 an hour, 3.07 times the H100 it would protect.

Prediction failures

Seventeen predictions were registered before the runs, each with a stated falsifier. Nine have verdicts so far and three of those were refuted. The failures are the useful part.

I predicted generation could be held exactly fixed. It cannot. One step in 80 produced 742 fewer tokens than the cap, with identical prompts, an identical seed, and a policy verified bitwise unchanged. The whole effect is 23 milliseconds across the ladder, so the curve stands and the phrase “exactly fixed” does not.

I predicted Modal’s cold start would land between 0.30 and 1.20 seconds. It measured 0.182. I had padded a published vendor figure with a margin I invented.

I predicted a 10 second reward would push idle past 75%. It reached 29%. I had sized the whole reward-time ladder against a table that assumed training was free, and training turned out to cost as much as generation. That result changed the experimental design.

One failure is not on that list, because the prediction survived. I predicted my synthetic reward was a fair stand-in, and the script testing it compared nominal service time when the prediction named measured. It printed REFUTED and was wrong. Corrected, the real-container effect is 1.3 points of idle beyond what service time explains, inside the bound I had set. The prediction held only because I found the bug in the code testing it.

That arm produced the more useful number anyway. A nominal 2 second sandbox reward measured between 2.45 and 8.64 seconds, with no tail injected at all. A reward’s service time is something you measure, not something you choose.

What this does and does not show

This is one model, one vendor, one GPU, and no convergence run. The reward is a controlled sleep rather than a real test suite, and the 30-second timeout is a configuration choice rather than something a real suite handed me. Several choices here push idle upward, so treat the absolute number as workload-specific. The barrier effect is the transferable result.

What it cost

About $7 in total, roughly two thirds of it on an idle GPU. Seventeen bugs, most of them in my own instrumentation rather than in the thing being measured.

The instrument failed more often than the subject did.

Summary

If you are moving from regular expressions to test suites, don’t start by optimizing the reward executor. Start by measuring the distribution of reward service times and how it meets your batch barrier.

Measure the bottleneck before you build infrastructure around it.

Harness, raw timing shards, and the pre-registered predictions with their verdicts: reward-latency-sweep.