Does an Opus agent run cost more than a Sonnet run on the same kind of work? 6x the cost, 3x the tool calls, 1.4x per self-estimated human hour

What it found

Joins 1,004 priced agent-run records (46 Opus, 958 Sonnet) and compares the models inside matched strata (same date and track, 11 strata with both), because pooled medians confound model with task mix. Matched, an Opus run costs 6.3× as much (95% bootstrap CI 4.7–8.0), makes 3.2× the tool calls, takes 6.3× the wall time and costs 2.0× per call, because it reads 1.9× more cached context per call. Against the run's own estimate of the human hours it replaces, Opus costs 1.4× more per hour (1.1–1.7). Costs are API-equivalent list-price figures, not charges; the human-hour estimate is the agent’s own and low-certainty; Opus runs are 46 and not randomised, so task difficulty can still differ inside a stratum. Preliminary: six days of data.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in The software-engineering loop

• Is an agent run getting more expensive day by day? Research runs grew about 5% a day in cost and tool calls, other tracks did notOn 1,053 priced run records (16 to 21 October 2026) the median research run grew about 4.9% a day in API-equivalent cost (95% interval +2.8 to +7.0) and 5.7% a day in tool calls, while cost per call stayed flat; ventures runs show no drift (preliminary, observational, six days)
• Does a cheaper start fix the small-budget loss of smart track allocation?Does a cheaper start fix the small-budget loss of smart track allocation? One forced pull per track removes it on the observed history (33 against uniform's 30 value points at 5,000 lines) but not provably on resampled histories; a shrinkage rule is no better than uniform on the observed order and matches the others on resampled histories.
• Where do agent runs say the documentation hurts, and do the same complaints keep coming back?Where do agent runs say the documentation hurts? 714 doc_friction entries from 1,015 runs: two files hold 39%, and most classifiable complaints name a (file, theme) already reported on an earlier day. Preliminary.
• Do documentation complaints in agent run records repeat more than chance?Do documentation complaints repeat more than chance? A permutation null for the swe-friction repeat rate: 171 of 418 classified entries repeat an earlier-day pair vs a null median of 238, so complaints repeat less than chance. Preliminary.
• Which test input should you use to calibrate a mass-spring-damper twin?Which test input should you use to calibrate a mass-spring-damper twin? A resonant sine gives the lowest held-out prediction error (3.1e-4 vs 1.1e-3 for a step) yet the worst Fisher condition number (527 vs 56 to 114); fitted spreads sit within 0.83 to 1.09 of the Cramér-Rao bounds. Stylised, preliminary.
• What can the run log measure about the agent loop today?What can the run log measure about the agent loop today? Cost fields are present in at most 1% of 189 runs, a third of listed projects are follow-up families, and every recorded CI-failure count is zero. Preliminary first step of track 10.

All 43 The software-engineering loop projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.