Proofload: The Diff Reproduced a Published Result on the First Run
Proofload is a statistical instrument for finding where an autonomous or agentic system breaks. You hand it a sweep spec and a rollout function of your own; it runs the grid, stores every episode, and tells you which cells moved between two runs and how small a move it could have seen. This is the build log, starting from an empty repo.
Sep 2: The dependency quarantine broke on day one
Scaffold, CI, five extras, eleven tests green. The core is meant to be numpy, pandas, scipy and pyarrow, with the survival-analysis reference implementations quarantined in a dev extra.
lifelines hard-depends on matplotlib. Also autograd, formulaic, narwhals, pillow, fonttools, kiwisolver and contourpy. The tripwire that checks the core pulls no extras passed only because nothing imported lifelines yet. Moved it to dev and hand-rolled Kaplan-Meier, Greenwood, Aalen-Johansen and log-rank in numpy and scipy, keeping lifelines as the oracle the tests compare against.
Small thing worth writing down: git add -A swept a symlink into a commit, because .gitignore’s notes/ pattern, with the trailing slash, matches directories only. A symlink is a file.
Sep 2: The publication rounded half up
The golden fixture is a Go1 quadruped push study converted into the store, so every later module has something real to run against. Two runs, 12,800 episodes each, 400 cells at 32 replicates.
Computed fell fractions came out at 62.7266% and 52.7578% against the published 62.7 and 52.8. Then the verification pass caught something else. The published boundary median for one cell is 101.3, the raw value is 101.25, and Python’s round gives 101.2. Same at two more cells. The publication rounded half up; Python rounds half to even. The boundary check now uses a half-decimal tolerance and says so in a comment.
Provenance was keyed on git rev-parse HEAD of the source repo, which means any commit over there, related or not, changes the hash a fresh build would write. Moved it to the last commit that actually touched the manifests. The two Parquet parts did not change a byte across the rebuild, which is a free confirmation that the determinism guarantee holds.
Sep 2: 60 safer, 0 worse, first run
The diff module went green 46 of 46 on its first run and reproduced the published result exactly: 60 cells significantly safer, none worse, per-cell Fisher exact with Benjamini-Hochberg correction across all 400 cells. Per-cell p and q match the publication to 1e-9.
I did not expect that. Every module before it needed at least one round trip.
Running the same episodes through the other two methods gives different counts: paired McNemar finds 55, log-rank finds 88. Neither is wrong. They are asking different questions of the same data, and the numbers are captured as a note for whoever writes the method documentation.
Sep 2: What “none worse” is worth
The worst-powered cell has a minimum detectable effect of 0.3436, at 32 replicates per arm and a base rate of 0.344. Across the grid that column runs from 0.0156 to 0.3436, median 0.0312.
So “no cell got worse” is an honest claim down to a 34-point rise in the worst cell, and a three-point rise would be missed in most of them. Every report the tool renders now prints that number beside the verdict count, because the verdict count is the thing a reader will quote and the other number is what makes it true.
Sep 2: An afternoon lost to a stale virtualenv
Renamed the package. The old name survives in four places where it is history rather than a claim.
Then I renamed the repo directory by hand, and source .venv/bin/activate quietly fell through to the global pyenv, because the activate script carries an absolute path. Consequences, all mine: an editable install landed in global site-packages, a pyarrow upgrade went there too, and the next session’s first acceptance run executed under the wrong interpreter with its plugins. Two mypy errors I spent real time on turned out to be the global pyarrow’s typing marker rather than anything I had written.
Rule from here. After a directory rename, rebuild the venv, and call .venv/bin/pytest directly instead of activating, so a stale path cannot fail silently.