Failure Boundary: The Gate That Kept Failing
Part 1 of mapping where a quadruped locomotion policy falls over. The plan: freeze a trained Go1 policy, sweep it across a grid of floor friction and lateral push, and measure survival per condition. Before any of that, the harness has to prove it isn’t lying.
Aug 13: Modal verification
Trained nothing yet. Spent the first sessions on scaffolding, CI, and confirming the GPU path works: A100-80GB, MJX physics, JAX. Recorded the budget and the gate order up front so I can’t quietly skip a check later when it’s inconvenient.
Aug 13: G2, the baseline policy
MuJoCo Playground’s Go1 joystick recipe, Brax PPO. 217.9M steps, 736 seconds on one A100. Nominal reward 30.30 ± 0.54. Walks clean, no falls.
That policy is now frozen. Everything downstream tests it; nothing retrains it.
Worth recording what it trained against, because it’s the honesty line for the whole study: friction randomized over U(0.4, 1.0), plus mass and center-of-mass jitter. No pushes. The environment ships a perturbation feature and it’s disabled by default; I left it disabled. So every shove in this study is out of distribution by construction.
Aug 13: G3, harness equivalence
The gate that matters. I inject a push as a body force and patch floor friction into the model. If that injection changes the physics in any way beyond the perturbation itself, every number afterward is measuring my wrapper instead of the policy.
Four Modal runs to get the gate design right, and the failures were the useful part.
Run 1. Designed an “exact” check: same seed, native environment vs injected-at-identity, tolerance 1e-4. It failed at 3.6e-4. The statistical gate passed. Tempting to relax the tolerance. Diagnosed instead.
Run 2. Added a replay check: step the wrapper from native states with identical actions and compare. Per-step state diffs came back exactly 0.0 over 500 steps. The wrapper is bitwise physics-identical.
The same run showed something else. Native reward at the same seed differed from run 1’s native reward: 28.8069 against 28.8105. Native versus native. Identical seed. That isn’t my wrapper, that’s the GPU.
Run 3. Bumped tolerance to 1e-3 provisionally. The same-seed diff spiked to 3.7e-3. Floating-point reduction order isn’t fixed on GPU, and a thousand steps of contact-rich dynamics amplify it into a broad distribution. Any tolerance on that comparison is a flaky gate by construction.
Run 4. Final structure. The gates are the bitwise replay check plus a statistical reproduction check. The same-seed comparison is demoted to a reported diagnostic, never a gate. Passed: replay 0.0, statistical |Δ| 0.481 against a 0.683 threshold.
What that cost me and what it bought
Four paid runs to arrive at a gate I could have written wrong and never noticed. The version I would have shipped first (tolerance on same-seed comparison) would have failed randomly forever and I’d have spent weeks blaming the wrapper.
It also produced the first real finding of the project, ahead of any actual result: the simulator does not agree with itself run to run. That has a consequence I didn’t expect yet but will keep running into. If a rollout isn’t reproducible, a single rollout can’t be evidence of anything.
Recorded a decision off the back of it: shareable-world determinism gets delivered by caching the first rollout as canonical, not by trying to force GPU recompute determinism. XLA deterministic-ops flags cost performance and stay fragile across hardware. The cache is honest about what it is.
Model facts pinned while I was in there: torso body trunk, robot mass 12.743 kg, g 9.81, so 100% of bodyweight is 125.0 N.