Bipedal Locomotion: The Gait-Phase Confound and Replication Audit
Part 2. Auditing the apparent superiority of checkpoint 1500 over converged 5999. See Part 1: Specimen Sizing and the Initial Sweep.
Sep 8: Reading the Zero-Variance Table
Proofload did not diagnose this. It ran the grid, stored every episode, and printed a table in which the absence of variance was impossible to miss: 8 cells, every one at exactly 0.0 or exactly 100.0, and a 72-cell developmental sweep the same night doing the same thing. The diagnosis took a person and about ten minutes.
The cause was in my rollout, not in the tool. The push was injected at control step 100, two seconds in, on every seed. The seed varied the initial pose by a few millimetres and nothing else that mattered. A biped’s response to a lateral shove depends on where it is in its stride when the shove lands, and with the push time fixed, every rollout in a cell was landing on the same stride phase. Checkpoint 1500’s gait happened to put a favourable phase at step 100.
The test wasn’t measuring policy robustness; it was measuring gait phase at an arbitrary timestamp.
Checkpoint 5999 under a 0.600 m/s impulse, control step 100 on the left and step 105 on the right. Same policy, same push, five control steps (0.10 s) apart. One falls, one does not.
Sep 8: Seed Conditioning across the Stride Cycle
A Fourier analysis of the steady walking gait put the stride period at about 20 control steps, 0.40 s at the 50 Hz control rate. So the push step is now drawn uniformly from 100 to 119, a full stride window (2.00 to 2.38 s), from the same seed Proofload hands the rollout. The stored episodes carry the push step in their metadata, so the next reader can check.
Re-ran 1,600 paired rollouts (8 cells, 100 seeds each, exact McNemar test with Benjamini-Hochberg FDR control at ).
Result: The apparent win for 1500 evaporated completely.
- At m/s (): 1500 had 20% failure; 5999 had 2% ().
- At m/s (): 1500 had 61% failure; 5999 had 12% ().
Model 5999 proved significantly safer across all 8 tested cells, with the largest across the family at . The worst-powered cell could detect an 18 to 20 pp change, so nothing smaller than that is claimed. The pseudo-replication bug was caught before publication.