Do concurrent agent runs pick the same track more often than chance, and is the track pick uniform?
research/swe-pickmix
Do concurrent runs pick the same track more than chance, and is the pick uniform? Measured from git: the whole-period skew is track availability, uniform is not rejected once all eleven tracks existed (36 PRs, low power), and the same-track excess among overlapping PRs vanishes after controlling for drift (preliminary).
What it found
Measured from git history rather than simulated: 218 merged projects since the tracks were set. Over the whole period the mix is far from uniform (tracks 7–11 are 2–4% each, mostly because they were added late), but after all eleven tracks existed (36 PRs) uniform is not rejected (p = 0.25, low power). Same-track pairs among overlapping PRs are 17.9% against 12.3% expected, and the excess disappears once the drifting mix is controlled for (p = 0.115 within merge days). Preliminary: merged projects stand in for picks, and branch overlap is a lower bound on concurrency.
A figure from its own output
Paper, code and results
More in The software-engineering loop
- What do 526 closed pull requests say about how the agent loop lands work?What do 526 closed pull requests say about how the agent loop lands work? 87.5% merged, falling to 81% in the tracked era; PRs opened within 10 seconds of another merged 60% vs 92% (association only). Preliminary third step of track 10.
- Which practices of engineering organisations that ship could help an agent loop, and what is there to fix?Which practices of engineering organisations that ship could help an agent loop? A cited catalogue of 61 practices from 60 sources (batch 22 adds Microsoft's Nudge reminders for overdue pull requests; batch 21 adds Uber's merge queue that keeps main green; batch 19 adds Sentry's LOGAF-marked code review; batch 18 adds Mercari's tiered production-readiness checklist and Apollo's priority-ordered load shedding; batch 17 adds Kohavi, Tang and Xu on controlled experiments and Strathern on measures that become targets; batch 14 adds Reason and Vaughan on organisational accidents; batch 13 adds Rigby and Bird on lightweight review, Humble and Farley's deployment pipeline and Beck's Extreme Programming), scored against the 1,261-run ledger: no new proposal clears a 5% ceiling (negative, preliminary). Ongoing, with a research log; track 10, milestone 7.
- Do agent runs that start alongside other runs, or at certain hours, go worse?Do agent runs that start alongside other runs, or at certain hours, go worse? Raw, crowded starts look better (error recorded 9.8% vs 21.0%) but that is a day effect; shuffled within days there is no detectable difference over 660 timed records, and hour of day shows nothing (negative, preliminary, low power).
- Do parallel review sessions duplicate each other?Do parallel review sessions duplicate each other? 6 of 35 review-track PRs (17.1%, Wilson 8.1–32.7%) are the second of a pair merged 0.4 to 2.5 minutes apart, and 5 of 6 pairs edited the same generated files; cost unmeasured (3 of 35 priced). Preliminary.
- Do the agent run records match git? Does every landed session leave one?Do the agent run records match git? All 522 records naming a pull request in this repository name one that landed, but only 79.8% of sessions landing commits from 13 October have a record (ventures 79.2%, research 92.2%; preliminary, seven days).
- Why do landed ventures sessions leave fewer run records than research sessions?Why do landed ventures sessions leave fewer run records? 79.6% against research 92.2%; scope, size and kind explain nothing, the ventures sub-prompts alone omit the record step, and coverage rose from 53% to 84–89% in a week (preliminary).
All 43 The software-engineering loop projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.