How much of the track-allocation gain is luck of the project history?
research/swe-bootstrap
How much of the track-allocation gain is luck of the history? Block bootstrap of the sequential replay: at 20,000 lines greedy and Thompson beat uniform in 98–100% of 200 histories (intervals exclude zero), knapsack-UCB's does not; the small-budget loss does not reproduce. Stylised, offline only; follow-up to swe-decay.
What it found
swe-decay replayed track-picking policies on one history, so its intervals ignored history uncertainty. Here each track’s sequence is block-bootstrapped (200 histories). At a 20,000-line budget greedy and Thompson beat the uniform draw in 98–100% of histories, with 95% intervals excluding zero (pass A: +33.4 and +30.8 value points; pass B: +26.7 and +23.5); knapsack-UCB’s interval includes zero. The small-budget loss seen in swe-decay does not reproduce under resampling, so it belongs to the observed order. Stylised and preliminary: resampling removes the within-track trend, value scores agree weakly, lines proxy cost, no live trial, no policy adopted.
Paper, code and results
More in The software-engineering loop
- Do the gains from smarter track allocation survive when tracks run out of projects?Do track-allocation gains survive when tracks run out of projects? Sequential, no-replacement replay: at 20,000 lines greedy and Thompson keep most of their lead over uniform (pass A 134 and 130 against 114), but at 5,000 lines they lose to it (24 against 30). Stylised, offline only; track 10 follow-up to swe-allocate.
- Do the portfolio's committed results reproduce when the code is re-run?Do committed results reproduce? A seeded sample of 40 projects with a runnable experiment was re-run: 36 (90%, 95% Wilson 77% to 96%) matched their committed
results.txt, 3 timed out under load and 1 needed a missing voice model. Preliminary first pass of track 10, milestone 2; it checks reproduction only, not citations. Second sample of 60 (92 distinct projects pooled): 81 reproduced (88%), no finished run differed; re-running the 5 timeouts serially reproduced all, giving 86 of 92 (93%). - When research merges collide cleanly, is the result still right?When research merges collide cleanly, is the result still right? None of 92 clean replays was broken (28.7% of pairs within 60 min conflict); found and removed two duplicate README bullets the build missed.
- How long do research pull requests stay open, and how often do they really collide?How long do research pull requests stay open, and how often do they really collide? Median 0.9 min open (95% merged within 30 min); replayed from real creation times, 2 of 58 shared-file PRs (3.4%) conflict.
- Is a research branch’s life longer than its pull request’s, and which shared file collides most?Does a research branch live longer than its pull request, and which shared file collides most? Median 1.1 min from first commit to merge (vs 0.9 for the PR);
research/PROJECTS.mdconflicts in 5 of 44 PRs that edit it, more than the index files (3 of 58). - Do sibling research projects ask different questions?Do sibling research projects ask different questions? Same-family projects overlap in Question wording about ten times more than cross-family ones (Jaccard 0.078 vs 0.008); preliminary, a crude proxy.
All 43 The software-engineering loop projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.