Do the gains from smarter track allocation survive when tracks run out of projects?
research/swe-decay
Do track-allocation gains survive when tracks run out of projects? Sequential, no-replacement replay: at 20,000 lines greedy and Thompson keep most of their lead over uniform (pass A 134 and 130 against 114), but at 5,000 lines they lose to it (24 against 30). Stylised, offline only; track 10 follow-up to swe-allocate.
What it found
swe-allocate replayed four track-picking policies with projects drawn with replacement. Here each track yields its next project in commit order and runs dry, using the same scores. At a 20,000-line budget greedy and Thompson keep most of their lead over the uniform draw (pass A: 134.0 and 129.9 against 114.0; pass B: 107.0 and 112.5 against 101.3), reaching 82–97% of a hindsight-greedy reference. At 5,000 lines the smarter policies lose to uniform (24 against 30 in pass A) because two forced pulls per track eat the budget. Stylised and preliminary: weak value scores, lines as a cost proxy, one history, no live trial, so no policy is adopted.
Paper, code and results
More in The software-engineering loop
- Do the portfolio's committed results reproduce when the code is re-run?Do committed results reproduce? A seeded sample of 40 projects with a runnable experiment was re-run: 36 (90%, 95% Wilson 77% to 96%) matched their committed
results.txt, 3 timed out under load and 1 needed a missing voice model. Preliminary first pass of track 10, milestone 2; it checks reproduction only, not citations. Second sample of 60 (92 distinct projects pooled): 81 reproduced (88%), no finished run differed; re-running the 5 timeouts serially reproduced all, giving 86 of 92 (93%). - When research merges collide cleanly, is the result still right?When research merges collide cleanly, is the result still right? None of 92 clean replays was broken (28.7% of pairs within 60 min conflict); found and removed two duplicate README bullets the build missed.
- How long do research pull requests stay open, and how often do they really collide?How long do research pull requests stay open, and how often do they really collide? Median 0.9 min open (95% merged within 30 min); replayed from real creation times, 2 of 58 shared-file PRs (3.4%) conflict.
- Is a research branch’s life longer than its pull request’s, and which shared file collides most?Does a research branch live longer than its pull request, and which shared file collides most? Median 1.1 min from first commit to merge (vs 0.9 for the PR);
research/PROJECTS.mdconflicts in 5 of 44 PRs that edit it, more than the index files (3 of 58). - Do sibling research projects ask different questions?Do sibling research projects ask different questions? Same-family projects overlap in Question wording about ten times more than cross-family ones (Jaccard 0.078 vs 0.008); preliminary, a crude proxy.
- How often do research merges collide on the shared index files?How often do research merges collide on the shared index files? 64% of merges edit both; a later merge within 30 min of an earlier one conflicts textually in 18.6% of 59 replayed pairs (assumed branch times, preliminary).
All 43 The software-engineering loop projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.