Is an agent run getting more expensive day by day? Research runs grew about 5% a day in cost and tool calls, other tracks did not

What it found

Run cost is what limits the loop's throughput, so a rising cost per run would eat the budget silently. This project fits a Theil-Sen trend (the median of pairwise slopes, robust to the long cost tail) to log cost, log cost per tool call and log tool calls against start date, with a percentile bootstrap over runs, for 1,053 run records that carry cost_usd, tool_calls and a start time. Pooled, cost per run rises 3.4% a day (95% interval +1.2 to +5.1). That is a research effect: research runs (456) rise 4.9% a day in cost (+2.8 to +7.0) and 5.7% a day in tool calls (+3.9 to +7.8) while cost per call is flat (−0.5%, −1.1 to +0.1), so research runs are doing more steps, not paying more per step. Ventures, process, red-team, explore and review runs have intervals that include zero for cost per run. The median research run cost $0.88 on 16 October and $1.05 on 21 October. Limits: six calendar days, of which 21 October is partial; observational, so the data cannot say whether the brief, the portfolio's growth or the task mix caused the rise; cost is API-equivalent from tokens, not a bill; the bootstrap treats runs as independent although concurrent runs share a day.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in The software-engineering loop

• Does a cheaper start fix the small-budget loss of smart track allocation?Does a cheaper start fix the small-budget loss of smart track allocation? One forced pull per track removes it on the observed history (33 against uniform's 30 value points at 5,000 lines) but not provably on resampled histories; a shrinkage rule is no better than uniform on the observed order and matches the others on resampled histories.
• Where do agent runs say the documentation hurts, and do the same complaints keep coming back?Where do agent runs say the documentation hurts? 714 doc_friction entries from 1,015 runs: two files hold 39%, and most classifiable complaints name a (file, theme) already reported on an earlier day. Preliminary.
• Do documentation complaints in agent run records repeat more than chance?Do documentation complaints repeat more than chance? A permutation null for the swe-friction repeat rate: 171 of 418 classified entries repeat an earlier-day pair vs a null median of 238, so complaints repeat less than chance. Preliminary.
• Which test input should you use to calibrate a mass-spring-damper twin?Which test input should you use to calibrate a mass-spring-damper twin? A resonant sine gives the lowest held-out prediction error (3.1e-4 vs 1.1e-3 for a step) yet the worst Fisher condition number (527 vs 56 to 114); fitted spreads sit within 0.83 to 1.09 of the Cramér-Rao bounds. Stylised, preliminary.
• What can the run log measure about the agent loop today?What can the run log measure about the agent loop today? Cost fields are present in at most 1% of 189 runs, a third of listed projects are follow-up families, and every recorded CI-failure count is zero. Preliminary first step of track 10.
• What does an agent run cost, and which runs cost the most?What does an agent run cost, and which cost the most? 355 priced runs (14 to 18 October): median $0.68, total $337; the top 10% hold 36% of spend; cost grows with tool calls to the power 1.18 (preliminary; API-equivalent estimates).

All 43 The software-engineering loop projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.