How often does this portfolio report a negative or mixed result?

research/swe-nullrate

How often does this portfolio report a negative or mixed result? In an 80-summary hand-labelled sample, 59% of the summaries that test a claim against an alternative (43–73%) report a negative or mixed result (preliminary, one labeller).

7 tests

What it found

A hand-labelled census of 80 randomly sampled project summaries (of 711): 51% descriptive, 20% positive, 18% mixed and 11% negative. Among the 39 that test a claim against an alternative, 59% (Wilson 95% 43–73%) report a negative or mixed result, so the portfolio is not a list of wins. A keyword rule for negative or mixed has precision 0.60 and recall 0.65 in-sample and should not be used for by-theme shares. Preliminary: one labeller, one pass, rubric fixed while labelling, summaries truncated when long, no inter-rater agreement.

A figure from its own output

Figure produced by research/swe-nullrate: headline_types — swe-nullrate/experiments/headline_types.svg, written by the project's own experiment code.

Paper, code and results

More in The software-engineering loop

Do later projects build on earlier ones?Do later projects build on earlier ones? A reference graph of 723 READMEs: 61.7% of projects are named by at least one later project (Wilson 58% to 65%), inbound counts are very uneven (Gini 0.69) and 71% of references stay within a track; naming is an upper bound on reuse.
Does an Opus agent run cost more than a Sonnet run on the same kind of work? 6x the cost, 3x the tool calls, 1.4x per self-estimated human hourDoes an Opus agent run cost more than a Sonnet run on the same kind of work? Matched on date and track, 6.3× the cost (CI 4.7–8.0), 3.2× the tool calls and 2.0× per call; 1.4× per self-estimated human hour. 46 Opus vs 958 Sonnet runs, preliminary.
Is an agent run getting more expensive day by day? Research runs grew about 5% a day in cost and tool calls, other tracks did notOn 1,053 priced run records (16 to 21 October 2026) the median research run grew about 4.9% a day in API-equivalent cost (95% interval +2.8 to +7.0) and 5.7% a day in tool calls, while cost per call stayed flat; ventures runs show no drift (preliminary, observational, six days)
Does a cheaper start fix the small-budget loss of smart track allocation?Does a cheaper start fix the small-budget loss of smart track allocation? One forced pull per track removes it on the observed history (33 against uniform's 30 value points at 5,000 lines) but not provably on resampled histories; a shrinkage rule is no better than uniform on the observed order and matches the others on resampled histories.
Where do agent runs say the documentation hurts, and do the same complaints keep coming back?Where do agent runs say the documentation hurts? 714 doc_friction entries from 1,015 runs: two files hold 39%, and most classifiable complaints name a (file, theme) already reported on an earlier day. Preliminary.
Do documentation complaints in agent run records repeat more than chance?Do documentation complaints repeat more than chance? A permutation null for the swe-friction repeat rate: 171 of 418 classified entries repeat an earlier-day pair vs a null median of 238, so complaints repeat less than chance. Preliminary.

All 43 The software-engineering loop projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.