Failure Boundary: The Survivor That Can't Be Re-Simulated
Part 3. The dataset exists, so now it needs to become something you can look at. This is where the nondeterminism from Part 1 stopped being a footnote and started dictating architecture. See also Part 2: 6,400 Rollouts.
Aug 14: The seed bug
Read the first render batch and all three “recovery” clips showed the robot falling.
Cause: replays and clips were seeding PRNGKey(seed_idx) directly, but the sweep’s scheme is PRNGKey(1_000_000 + rollout_index). The artifacts were simulating entirely different worlds that happened to share a label.
The fix: sweep/seed_scheme.py, the seed scheme as a pure importable fact, tested against both the executor’s constants and the committed sweep records. Every replay and clip consumer derives seeds from there now.
Cost of that mistake: one full render batch on mislabeled seeds, about 57 minutes of GPU.
Aug 14: The finding underneath the bug
With the exact correct seeds, the sweep’s recorded survivors in the three rarest cells still fell. So I searched all 16 exact seeds of each cell.
Zero survivors on re-simulation.
That’s statistically coherent. At a survival probability around 0.06, an empty 16-trial re-run has probability about 0.36 per cell, before you even account for GPU nondeterminism. But it’s decisive for the design:
The sweep’s rarest survivor is provably real, since its summary tracks are committed. It is also provably un-regenerable. Ask the same world for that same survivor again and you get a fall.
A survivor at the edge: minimum uprightness 0.66 and minimum torso height 0.148 m, both a hair above the fail thresholds. This one re-materialized. Several didn’t.
Replays of knife-edge rollouts exist in the cache or they do not exist. That’s the strongest argument I have for making the cache the canonical record rather than a performance optimization.
Design responses:
- Recovery selection now requires re-findable survivors. Cells with survival between 3/16 and 0.5, one recovery per cell.
- The replay batch searches the cell’s exact sweep seeds, recorded-survivor first, for one that survives now, and records the search. Replay honesty is “a surviving rollout of this world,” never a mislabeled world.
Aug 15: Clip equals replay, by construction
Second review round. The fail clip didn’t fall. Again.
Root cause, and it’s a real defect rather than a rendering issue: clips and replays each re-simulated the same seed independently. Comparing the two across the 18 selected worlds, 10 of 18 outcomes disagreed, including all three recoveries. The clip’s “fail” label came from selection metadata while its own re-simulation had survived.
Decision: outcomes are decided once, by the replay batch, and clips render the batch’s saved trajectory. The qpos gets persisted beside each .rrd. Clip equals replay equals cached artifact, by construction, never an independent re-simulation.
Re-running the replay batch found no survivor in two of the three recovery cells this time. Previous batch found one in each. That’s the third independent measurement of the same phenomenon and it goes in the writeup.
The floor
The trimmed fall clip. It stops at the frame the torso reaches fail height.
“The robot falls through the floor” turned out not to be a render bug. The Playground Go1 MJX model is feet-only collision. Body-ground contact does not exist in the model the policy trained in.
So clips freeze on the frame the torso reaches fail height. The topple reads; the sink is never shown. It’s a property of the training simulator and it goes in the post rather than getting quietly cropped.
Aug 15: Scope decision
Combined the flagship with what was going to be a separate case study. The post now ships the full loop: map the frozen policy, make one deliberate training-distribution change, retrain, re-map, diff the surfaces cell by cell.
Two heatmaps and a diff is a stronger result than one heatmap. It also means both checkpoints end up live-probeable.
Guards recorded before running anything: equivalence re-run per checkpoint is mandatory, and envelope shrinkage gets reported anywhere it appears. If the retrain makes something worse I need that in the post, not discovered by a reader.