Which track should the next run work on, and does a smarter pick beat a uniform one?

What it found

Four track-allocation policies (uniform, greedy value per line, bootstrap Thompson sampling, and a knapsack-style UCB) were replayed offline on 7 tracks, using the 408-project scores from swe-value (value 1 to 3, cost in lines added) under a 20,000-line budget and 300 seeds. Greedy and Thompson reach about 79–83% (pass A) and 75–79% (pass B) of the best-single-track oracle, against 58–60% for the current uniform pick; the UCB variant explores too much to help (62–66%). This is a stylised, preliminary result: arms are assumed stationary, yet value per line falls with order in 4 of 7 tracks (Spearman −0.36 to −0.57), the value scores themselves agree only weakly, lines are a poor proxy for tokens, and no live trial was run, so no policy is adopted.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in The software-engineering loop

• How much of the track-allocation gain is luck of the project history?How much of the track-allocation gain is luck of the history? Block bootstrap of the sequential replay: at 20,000 lines greedy and Thompson beat uniform in 98–100% of 200 histories (intervals exclude zero), knapsack-UCB's does not; the small-budget loss does not reproduce. Stylised, offline only; follow-up to swe-decay.
• Do the gains from smarter track allocation survive when tracks run out of projects?Do track-allocation gains survive when tracks run out of projects? Sequential, no-replacement replay: at 20,000 lines greedy and Thompson keep most of their lead over uniform (pass A 134 and 130 against 114), but at 5,000 lines they lose to it (24 against 30). Stylised, offline only; track 10 follow-up to swe-allocate.
• Do the portfolio's committed results reproduce when the code is re-run?Do committed results reproduce? A seeded sample of 40 projects with a runnable experiment was re-run: 36 (90%, 95% Wilson 77% to 96%) matched their committed results.txt, 3 timed out under load and 1 needed a missing voice model. Preliminary first pass of track 10, milestone 2; it checks reproduction only, not citations. Second sample of 60 (92 distinct projects pooled): 81 reproduced (88%), no finished run differed; re-running the 5 timeouts serially reproduced all, giving 86 of 92 (93%).
• When research merges collide cleanly, is the result still right?When research merges collide cleanly, is the result still right? None of 92 clean replays was broken (28.7% of pairs within 60 min conflict); found and removed two duplicate README bullets the build missed.
• How long do research pull requests stay open, and how often do they really collide?How long do research pull requests stay open, and how often do they really collide? Median 0.9 min open (95% merged within 30 min); replayed from real creation times, 2 of 58 shared-file PRs (3.4%) conflict.
• Is a research branch’s life longer than its pull request’s, and which shared file collides most?Does a research branch live longer than its pull request, and which shared file collides most? Median 1.1 min from first commit to merge (vs 0.9 for the PR); research/PROJECTS.md conflicts in 5 of 44 PRs that edit it, more than the index files (3 of 58).

All 43 The software-engineering loop projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.