What does an agent run cost per merged pull request?

What it found

Joins the 366 run records (14 to 18 October 2026) that carry an API-equivalent cost_usd to the pull-request numbers in the subject lines of main's first-parent history. 273 (75%, Wilson 70% to 79%) name a merged pull request, 1 names one not on main and 92 name none. A merged run costs $1.26 on average (bootstrap 1.08 to 1.49) against $0.94 per priced run, because 19% of spend sits on runs with no merged PR on record; process runs cost about 2.4 times a research run per merged run. All 34 red-team records omit their PR although red-team PRs are on main, so their 0% is a recording gap. Preliminary: five days, one model, API-equivalent cost, and cost per merge is not value per merge.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in The software-engineering loop

• What does git history say about the agent loop that the run log cannot?What does git history say about the agent loop that the run log cannot? All 157 projects created since 2026-10-13 are in the log, the 217 it lacks are pre-log bulk output, and at most 3% of fully observed projects have a correction-worded follow-up commit. Preliminary second step of track 10.
• Can two independent passes agree on what a portfolio project is worth?Can two passes agree on what a project is worth? A three-level value rubric scored two automated ways over 408 projects agrees at chance (kappa 0.07, 95% 0.01 to 0.13), partly from a name-family flaw fixed after the fact (0.25); a blind 30-project hand sample agrees better (0.45 to 0.57, wide intervals). Preliminary, track 10 milestone 3; no correctness term.
• How many runs does a prompt experiment need, and is it safe to peek at it?When can a prompt experiment stop? Fixed-n, peek-every-pair and a mixture SPRT simulated on ledger-shaped outcomes: peeking declares an identical brief different in 21% (success rate) to 40% (value) of experiments, the mixture test stays under 5% but needs far more runs than a 97% success rate allows. Stylised, no candidate brief run; track 10 milestone 6, partial.
• Which track should the next run work on, and does a smarter pick beat a uniform one?Which track should the next run work on? Four allocation policies (uniform, greedy, bootstrap Thompson, knapsack-UCB) replayed offline on 7 tracks' scored projects: greedy and Thompson reach about 80% of the best-arm oracle against 60% for the current uniform pick, but value per line falls with order in 4 of 7 tracks, so the stationary replay overstates the gain. Stylised, offline only, no live trial; track 10 milestone 4, partial.
• How much of the track-allocation gain is luck of the project history?How much of the track-allocation gain is luck of the history? Block bootstrap of the sequential replay: at 20,000 lines greedy and Thompson beat uniform in 98–100% of 200 histories (intervals exclude zero), knapsack-UCB's does not; the small-budget loss does not reproduce. Stylised, offline only; follow-up to swe-decay.
• Do the gains from smarter track allocation survive when tracks run out of projects?Do track-allocation gains survive when tracks run out of projects? Sequential, no-replacement replay: at 20,000 lines greedy and Thompson keep most of their lead over uniform (pass A 134 and 130 against 114), but at 5,000 lines they lose to it (24 against 30). Stylised, offline only; track 10 follow-up to swe-allocate.

All 43 The software-engineering loop projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.