Failure Boundary: The Retrain Moved It

August 18, 2026

Part 5, the second act. The map exists and is probeable; now it has to be worth having made. See Part 4: Cold Starts and Host Pointers.

Aug 18: The API

app/core.py as a pure, type-checked brain with 17 contract tests: validation and clamping including hostile input, cache-hit path, degradation mode, per-IP and global limits, spend ledger with day rollover, and analytics that record no identity. app/server.py is a thin Modal ASGI shell over it.

Verified live: a cached world returns instantly at zero cost with its canonical artifact URL. An uncached world runs a real GPU probe through the public API, 30.7 s wall on one snapshot restore. Hostile input clamps and says so.

Deploy lessons, both self-inflicted:

• Blocking Modal calls inside an async handler stalled the event loop and Modal cancelled the input. Fixed with an async body read plus a threadpool handoff.
• from __future__ import annotations made FastAPI treat Request as a query parameter and return 422.

One decision I want to keep visible: unparseable friction maps to None, meaning model default, not 0.0. The first deploy did the latter, which silently sent bad input to the ice floor. Wrong semantics, caught in live testing, test added.

Aug 18: The one change

v1 is the exact same recipe as the baseline with one thing different: the training distribution. Environment velocity kicks on, friction randomization floor widened from 0.4 down to 0.1. Both were chosen by the map, since those are precisely the two regions where it goes dark.

The delta is recorded verbatim with source hashes, so “we changed exactly two things” is checkable rather than claimed.

Retrain: 751 s.

Aug 18: The gate failed, and the gate was wrong

v1’s first equivalence check failed. Statistical gate: our nominal harness eval at 23.6 against v1’s training eval at 19.7.

Then I looked at what those two numbers actually are. The training eval runs with kicks enabled, because that’s what v1 trains under. The sweep runs nominal. That’s apples to oranges for any perturbation-trained checkpoint, and it would fail for every future checkpoint of that kind.

Fixed the gate rather than the result: it now compares native harness against injected harness, both nominal, over K repeats. That’s checkpoint-agnostic and it’s the actual question, which is whether the executor is faithful to the plain environment.

v1 then passed cleanly. Replay bitwise 0.0, matched injected 23.66 against native 23.94, |Δ| 0.28 inside a 0.76 gate. The training-eval gap is recorded as context, not as a failure.

This is the second time on this project that a gate failure turned out to be a gate design problem rather than a real defect. Both times the fix made the gate correct for cases I hadn’t run yet.

Aug 18: The result

Both checkpoints re-swept at 32 seeds per cell. Per-cell Fisher exact with Benjamini-Hochberg correction across all 400 cells, plus bootstrap confidence intervals on the boundary.

60 cells significantly safer. None significantly worse.

The boundary moved outward on every friction row. Gains are largest exactly where I intervened:

Same world (μ=0.60, 90% bodyweight push) on both policies. Baseline left, retrained right. Representative rollouts; seeds differ per side.

The shrinkage census is empty. I set that guard up before running it specifically so I couldn’t quietly not look.

GPU ledger for the session: v1 smoke ~5 min, v1 full ~13, v0 re-sweep ~4, v1 re-sweep ~4, two equivalence runs ~8. About 34 A100-minutes.

Where it landed

Whole study, training through the retrain and the second full sweep, came to $42 of compute off the bill.

That number is the one I keep coming back to. Not because it’s impressive on its own, but because at that price the map stops being a thing you make once for a writeup and becomes something you can run against every checkpoint.