How well does a run's tool-call count predict what it costs?

research/swe-callcost

How well does a run's tool-call count predict its cost? On 882 priced run records cost is about $0.03 per call, a log-log fit has R² 0.91 and a held-out median error of 9%; the pooled slope 1.22 is partly group mixing (1.10 within Sonnet). Preliminary.

8 tests

What it found

Of 1,638 run records, 882 carry both a cost and a tool-call count. Cost per call is almost constant (median $0.031, 10th to 90th percentile $0.026 to $0.038); a log-log fit has R² 0.91 and misses held-out runs by a median of 9% (90th percentile 25%), no better than a flat cost-per-call multiplier. The pooled slope of 1.22 (1.18 to 1.26) is partly group mixing: it is 1.10 within Sonnet runs and 0.96 within Opus runs (wide interval). Cost is API-equivalent and derived from the same transcript as the call count, so the fit checks the measurement, not agent behaviour. Preliminary.

A figure from its own output

Figure produced by research/swe-callcost: cost_vs_calls — swe-callcost/experiments/cost_vs_calls.svg, written by the project's own experiment code.

Paper, code and results

More in The software-engineering loop

Where does an agent run's cost go, and why does it grow faster than its tool calls?Where does a run's cost go? On 893 run records with per-token-kind usage, cache reads are a median 45% of the API-equivalent cost and 1-hour cache writes 41%, output only 13%; the starting context (about 60,000 tokens) is re-read on every request, which makes cost grow faster than linearly in tool calls (preliminary).
Do agent runs that add no tests end worse?Do agent runs that add no tests end worse? 84 of 621 counted run records add none; within track they end partial 4/84 against 16/537 (+2.6 points, p = 0.23, no detectable effect) and cost less, probably because they do less. Preliminary.
Which kind of recorded deviation from the brief predicts a partial outcome?Which kind of recorded deviation predicts a partial outcome? Of 213 run records with a deviation, tooling failures end partial 22.6 points more often than runs with none (track-stratified p=0.001); skipped steps (+1.3) and switched tasks (+1.3) do not. Keyword classes, observational, preliminary.
How complete are the agent run records, field by field?How complete are the agent run records, field by field? Of 1,362 records, cost, tokens and wall time are present on 47% (0-7% on 13 to 15 October, 61-82% since), merged on 4.6%, and 18.5% of success runs name no pull request (preliminary; presence, not correctness).
Does a bigger agent run buy more output?Does a bigger run buy more output? Across 157 priced run records that report adding tests, cost and tests added rank together pooled (Spearman 0.57) but weakly within research runs (0.19), and the dearest third yields the fewest tests per dollar; preliminary, tests added is a crude self-reported proxy.
What does an agent run cost per merged pull request?What does a run cost per merged pull request? 366 priced runs joined to main's merge history: 75% name a merged PR, $1.26 per merged run against $0.94 per run, process runs about 2.4 times research's; red team never links its PR (preliminary; API-equivalent estimates).

All 43 The software-engineering loop projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.