Failure Boundary: Cold Starts, Host Pointers, and Hitting the Spend Limit

August 18, 2026

Part 4. Making the map probeable live, which means a GPU that wakes up fast enough that nobody closes the tab. This was the most expensive day of the project and most of it went on one bug. See Part 3: The Survivor That Can’t Be Re-Simulated.

Aug 18: First latency measurement

Deployed the live probe. Same physics path as the replay batch, same artifact out. JAX persistent compilation cache warmed at image build by running the exact probe program on the serving GPU class, since the cache key includes the GPU name.

Five cold starts, twenty warm calls:

Warm passes. Cold misses badly.

The warm breakdown was the surprise: physics is 0.52 s. The Rerun logging and save is 4.3 s. The thing dominating my warm path was CPU work, not the GPU.

Aug 18: Lever 1, columnar logging

Replaced roughly 7,500 individual rr.log calls with send_columns per entity.

Warm p50 went 7.2 s to 1.8 s. Logging went 4.3 s to 0.2 s. Warm is now Modal’s request overhead rather than my code, which is where I want it.

Cold was unchanged, because cold was never about logging.

Aug 18: Lever 2, autotune off

Turned XLA GPU autotuning off. Build-time compile fell about 15 s. Genuinely cold containers still spent 18 to 21 s inside “sim.”

Reverted. It bought nothing at runtime and gave up an optimization.

Study note worth keeping: one row in that batch showed a 0.5 s “cold” sim, which was a container that hadn’t actually scaled down in the 75 s idle window. Real colds are the 18 s and up rows. Easy way to fool yourself.

Aug 18: Lever 3, the actual root cause

Instrumented the serving process properly. JAX cache logger, baked-key listing, key-component hashes, HLO dumps from two separate processes.

• Small programs hit the persistent cache fine.
• jit(rollout) misses the cache on every single cold start. Fourteen to twenty-one seconds of compile, and it writes a different key every process. I counted at least four keys for identical code.
• Diffing the key components across two cold processes: XLA flags, backend, compile options, accelerator config, jax_lib all identical. The computation component differs.
• So I diffed the HLO text. 4,231 lines each, 12 lines different. All of them int32 constants of shape 18, 40, 14, 12, whose values are 64-bit host memory pointers split into halves. High words 22034 against 22091. Those are per-process heap addresses.

Something in the traced pytree, almost certainly bookkeeping leaves of the mjx Model, is carrying host pointers into the compiled program as constants. New pointers per process, new HLO hash per process, cache miss forever.

Changing params from a closure to a jit argument did help a bit: the baked jit_rollout keys went from 2 to 1. Kept, but it doesn’t fix the model leaves.

Aug 18: The halt

“Container terminated due to reaching billing cycle spend limit,” mid-deploy, during the audit that would have identified the offending leaves.

Honest accounting on why: this session ran roughly eight image builds, each with a GPU warm-up at about 90 s of A100, plus five latency studies at around ten minutes each. My running estimate had been under-counting the GPU time inside build steps. Largest single-day spend of the project by a wide margin.

The guard did its job. Nothing was lost: every measurement is committed, the audit instrumentation is committed, and the diagnosis is complete and specific. What remained was one fix and one re-measure.

Aug 18: Memory snapshot, which actually worked

Enabled GPU memory snapshotting with the JIT and warm-up inside the snapshot phase. One-time snapshot creation cost 165.8 s.

On restored containers, sim is 0.51 s and the function total is 0.7 to 1.6 s. The compile is gone. All remaining cold time is snapshot restore plus Modal scheduling, none of it my code.

Cold still misses the under-20 s target. But the residual is entirely platform, and the honest framing for the writeup is better than a number that passes: a cold probe is about 24 s of platform restore plus one second of physics, and during launch week I keep one container warm and say so.