Where a 14-DoF Biped Falls
Originally published at Poisson Labs.
This is the short version. The full post, with the per-cell table, the developmental sweeps and every linked Proofload report, is on Poisson Labs.
I trained a walking policy for Microduck, Pollen Robotics’ 14-degree-of-freedom open-source biped, saved a checkpoint every 250 iterations, and used Proofload to measure where each frozen checkpoint falls under a lateral push.
Proofload is the statistical instrument I’ve been building at Poisson Labs. It takes a sweep spec and an import path to a rollout function of your own; the function gets one condition and one seed and returns an outcome and a time. The tool builds the grid, derives paired seeds from a run salt, runs the rollouts, stores every episode, computes survival per cell, and for two runs computes the paired difference per cell with a McNemar exact test, Benjamini–Hochberg correction across the grid, and the minimum detectable effect each cell had. Nothing in that list knows what a robot is. The physics lives in my rollout function.
The seeds weren’t varying anything
The first paired run went into the store a little after eight on the evening of September 8, and it said checkpoint 1500 was the robust one: 0 falls in 800 rollouts, against 400 in 800 for the converged checkpoint 5999. The paired report made that hard to believe. Every cell was exactly 0% or exactly 100%, and so was every cell of a 72-cell developmental sweep run the same night. A hundred seeds per cell and no within-cell variance.
The push was landing on control step 100 in every rollout, so every seed hit the same stride phase. Checkpoint 1500’s gait happened to have a good phase at step 100. I made the push step a seeded draw over a 20-step stride window, reran the same 8 cells twenty-four minutes later, and it flipped: checkpoint 5999 significantly safer in 8 of 8, largest q 0.0027, worst-cell MDE 20 pp.
Proofload didn’t diagnose this. It ran the grid, kept every episode, and printed a table where the missing variance was obvious. Reading it was my job.
A cliff, a slope, and a crossover
With the phase fixed, I resolved checkpoint 2250 against checkpoint 5999 from 0.550 to 0.700 m/s in 0.025 m/s steps, two frictions, 100 paired rollouts per cell.
Checkpoint 2250 has a cliff: 2 falls in 600 rollouts at or below 0.600 m/s, then 52 to 58% at 0.625, 96 to 99% at 0.650, and every rollout from 0.675 up. Checkpoint 5999 has a slope: 5 to 16% below the cliff, 16% at 0.625, and between 25% and 45% from 0.650 to the edge of the grid. At or below 0.600 m/s, 2250 wins and Proofload says so in five of six cells. From 0.625 m/s up, 5999 wins in all eight. Worst-cell MDE 19 pp. Runs 20260909T010313Z-5abe and 20260909T010448Z-a6f1.
A single benchmark push can’t tell you which shape you have, and which one you want depends on how sure you are about the disturbances you’ll actually meet.
The retrain made it worse
The next morning I tried to widen the slope. The upstream repo ships a VelStand task that adds stumble recovery and getting back upright to the walking objective; a policy that had learned to recover ought to hold a wider boundary. I trained it with the same seed and iteration count and diffed its converged checkpoint against the baseline’s on 14 cells from 0.600 to 0.750 m/s.
Worse in 14 of 14 cells, by 19 to 57 percentage points; at 0.600 m/s the fall rate goes from 12% to 62% in both friction rows. Largest q 0.0054, worst-cell MDE 20 pp. Runs 20260909T102454Z-1090 and 20260909T102639Z-21a3. A developmental sweep along both training runs shows the two recipes pass through the same rigid checkpoint and lose it at the next one sampled; the baseline then works its fall rate at 0.600 m/s down from 66% to 15%, and VelStand ends at 63%. I don’t have a measured mechanism for why, and the full post says so.
The receipt
Everything ran locally on one Apple Silicon laptop: 28,920 rollouts across 12 stored runs in about 31 minutes of wall time, summed from the per-episode times in the store. Training was on Colab Pro+ with an L4, about 4 h 45 min for the baseline; I didn’t record the dollar figure. Envelope: push 0.45 to 0.75 m/s, friction 0.6 and 1.0, a 500-step horizon at 50 Hz, 100 rollouts per cell. The worst-powered cell in any diff can detect a 20 pp change, and I don’t claim anything smaller than that anywhere.
The full post is at poissonlabs.ai/research/map-the-failure-boundary-microduck. Proofload is open source, releasing soon. If you train policies and want to know where they break before someone else finds out, write to work@poissonlabs.ai.
Built on Proofload, MuJoCo, mjlab, microduck_rl (Pollen Robotics / Hugging Face Microduck), and rsl_rl.