Do the portfolio's committed results reproduce when the code is re-run?

What it found

A seeded sample of 40 research/ projects with a runnable experiment was re-run on a clean copy and diffed against the committed results.txt. 36 of 40 (90%, 95% Wilson interval 77% to 96%) reproduced: 34 byte-identical and 2 differing only in a timing line. Three timed out at 300 s (four jobs in parallel) and one needs a voice model that is not in the repository, so those four are inconclusive, not failures. Of 713 decimal numbers in the sampled READMEs, 85% appear verbatim in committed result files (the rest are derived or in prose). A second sample of 60 (8 overlapping) brought the pooled total to 92 distinct projects: 81 reproduced (88%, Wilson 80% to 93%), none of the 81 finished runs gave different numbers, and the 11 others are missing tools (Yosys, MuJoCo, a voice model) or timeouts. Re-running the five timeouts one at a time (no parallel load) reproduced all five (two differing only in timing lines), lifting the pooled figure to 86 of 92 (93%, Wilson 86% to 97%); the 6 left are missing tools or the voice model. A mechanical reproduction check on a small sample, with no citation audit, so the correctness term is only partly estimated. Preliminary.

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in The software-engineering loop

• When research merges collide cleanly, is the result still right?When research merges collide cleanly, is the result still right? None of 92 clean replays was broken (28.7% of pairs within 60 min conflict); found and removed two duplicate README bullets the build missed.
• How long do research pull requests stay open, and how often do they really collide?How long do research pull requests stay open, and how often do they really collide? Median 0.9 min open (95% merged within 30 min); replayed from real creation times, 2 of 58 shared-file PRs (3.4%) conflict.
• Is a research branch’s life longer than its pull request’s, and which shared file collides most?Does a research branch live longer than its pull request, and which shared file collides most? Median 1.1 min from first commit to merge (vs 0.9 for the PR); research/PROJECTS.md conflicts in 5 of 44 PRs that edit it, more than the index files (3 of 58).
• Do sibling research projects ask different questions?Do sibling research projects ask different questions? Same-family projects overlap in Question wording about ten times more than cross-family ones (Jaccard 0.078 vs 0.008); preliminary, a crude proxy.
• How often do research merges collide on the shared index files?How often do research merges collide on the shared index files? 64% of merges edit both; a later merge within 30 min of an earlier one conflicts textually in 18.6% of 59 replayed pairs (assumed branch times, preliminary).
• How many parallel agent runs duplicate each other when they start in the same minute, and what removes the waste?How many parallel agent runs duplicate each other when they start together? 12.8% of runs at four blind starts; a start gap past the claim latency or scheduler slots remove it (stylised, assumed timings).

All 43 The software-engineering loop projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.