Did research brief v1 change how runs go?
research/swe-briefab
Did research brief v1 change how runs go? Same-days comparison of v0 (342 runs) and v1 (67) on 15 to 18 October: no detectable difference in cost, time or success; the lower v1 error rate (7.5% vs 17.0%) vanishes within unlabelled-task runs (8.3% vs 9.6%); v1 records more doc-friction entries (1.24 vs 0.72 per run, p = 0.0001). Observational.
What it found
Research runs on prompt v0 (342) and v1 (67) from the same four days, 15–18 October 2026. Cost, wall time and output tokens do not differ detectably (median cost $0.835 vs $0.820, p = 0.72). v1 records fewer errors (7.5% vs 17.0%, p = 0.058), but v0 drew far more extend and new-project tasks, and among runs with no task label the rates are 8.3% vs 9.6%. v1 runs list more documentation friction (1.24 vs 0.72 entries per run, p = 0.0001; 1.28 vs 1.02 among unlabelled runs). Observational and preliminary: task labels differ between arms and seven metrics were tested uncorrected.
Paper, code and results
More in The software-engineering loop
- Does Opus pay for itself on this portfolio's runs? Cost ratio by track, and the sample needed to find outDoes Opus pay for itself on this portfolio's runs? An Opus run costs about 8 to 9 times a Sonnet run in the same track; with 12 priced Opus runs the ledger cannot compare value per dollar, and a 50% difference in projects per run needs about 129 runs per arm (preliminary; cost only).
- Do agent runs that record a deviation from the brief cost more or end worse?Do agent runs that record a deviation from the brief cost more or end worse? Of 1,564 run records 203 list one; within track they take ~1.6x as long and cost ~1.2x as much, but the cost gap rests mainly on the process track, and they end partial 5 points more often. Observational, preliminary.
- How well does a run's tool-call count predict what it costs?How well does a run's tool-call count predict its cost? On 882 priced run records cost is about $0.03 per call, a log-log fit has R² 0.91 and a held-out median error of 9%; the pooled slope 1.22 is partly group mixing (1.10 within Sonnet). Preliminary.
- Where does an agent run's cost go, and why does it grow faster than its tool calls?Where does a run's cost go? On 893 run records with per-token-kind usage, cache reads are a median 45% of the API-equivalent cost and 1-hour cache writes 41%, output only 13%; the starting context (about 60,000 tokens) is re-read on every request, which makes cost grow faster than linearly in tool calls (preliminary).
- Do agent runs that add no tests end worse?Do agent runs that add no tests end worse? 84 of 621 counted run records add none; within track they end partial 4/84 against 16/537 (+2.6 points, p = 0.23, no detectable effect) and cost less, probably because they do less. Preliminary.
- Which kind of recorded deviation from the brief predicts a partial outcome?Which kind of recorded deviation predicts a partial outcome? Of 213 run records with a deviation, tooling failures end partial 22.6 points more often than runs with none (track-stratified p=0.001); skipped steps (+1.3) and switched tasks (+1.3) do not. Keyword classes, observational, preliminary.
All 43 The software-engineering loop projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.