Do the agent run records match git? Does every landed session leave one?

What it found

Checks the run records against the commit history on main. All 522 records that name a pull request in this repository name one that landed (Wilson 95% 99.3–100%). But only 79.8% of sessions that landed commits from 13 October have a record (1,465 of 1,836; 77.9–81.6%): 89.8% of sessions that followed the master prompt, 4.2% of those that did not, and by track research 92.2%, process 93.2%, red team 100% and ventures 79.2% (75.5–82.5%). Coverage rose from about 60% on 13–14 October to 82–88% afterwards. Preliminary: seven days; sessions are identified by a commit trailer, so sessions without one are invisible; landed is a git fact, not a quality one.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Code
• Tests

More in The software-engineering loop

• Why do landed ventures sessions leave fewer run records than research sessions?Why do landed ventures sessions leave fewer run records? 79.6% against research 92.2%; scope, size and kind explain nothing, the ventures sub-prompts alone omit the record step, and coverage rose from 53% to 84–89% in a week (preliminary).
• How often does this portfolio report a negative or mixed result?How often does this portfolio report a negative or mixed result? In an 80-summary hand-labelled sample, 59% of the summaries that test a claim against an alternative (43–73%) report a negative or mixed result (preliminary, one labeller).
• Do later projects build on earlier ones?Do later projects build on earlier ones? A reference graph of 723 READMEs: 61.7% of projects are named by at least one later project (Wilson 58% to 65%), inbound counts are very uneven (Gini 0.69) and 71% of references stay within a track; naming is an upper bound on reuse.
• Does an Opus agent run cost more than a Sonnet run on the same kind of work? 6x the cost, 3x the tool calls, 1.4x per self-estimated human hourDoes an Opus agent run cost more than a Sonnet run on the same kind of work? Matched on date and track, 6.3× the cost (CI 4.7–8.0), 3.2× the tool calls and 2.0× per call; 1.4× per self-estimated human hour. 46 Opus vs 958 Sonnet runs, preliminary.
• Is an agent run getting more expensive day by day? Research runs grew about 5% a day in cost and tool calls, other tracks did notOn 1,053 priced run records (16 to 21 October 2026) the median research run grew about 4.9% a day in API-equivalent cost (95% interval +2.8 to +7.0) and 5.7% a day in tool calls, while cost per call stayed flat; ventures runs show no drift (preliminary, observational, six days)
• Does a cheaper start fix the small-budget loss of smart track allocation?Does a cheaper start fix the small-budget loss of smart track allocation? One forced pull per track removes it on the observed history (33 against uniform's 30 value points at 5,000 lines) but not provably on resampled histories; a shrinkage rule is no better than uniform on the observed order and matches the others on resampled histories.

All 43 The software-engineering loop projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.