How many runs does a prompt experiment need, and is it safe to peek at it?

research/swe-prompt

When can a prompt experiment stop? Fixed-n, peek-every-pair and a mixture SPRT simulated on ledger-shaped outcomes: peeking declares an identical brief different in 21% (success rate) to 40% (value) of experiments, the mixture test stays under 5% but needs far more runs than a 97% success rate allows. Stylised, no candidate brief run; track 10 milestone 6, partial.

7 tests

What it found

Three stopping rules for a champion-versus-candidate brief experiment were simulated on outcome distributions drawn from the run ledger (success rate 237/244) and from swe-value’s project values: one look at the end, a look after every pair, and a mixture sequential probability ratio test. Looking after every pair declared an identical brief different in 21% of 2,000 experiments on success rate and 40% on value (nominal 5%); the mixture test stayed at 0.4% and 3.9%, at the price of power (92% against 99% at a +0.2 shift in value, stopping after 181 of 400 pairs on average). At a 97% success rate a fixed test needs about 2,200 runs to see a 0.971 to 0.991 change, so success rate is a poor primary metric. Stylised and preliminary: all 244 ledger rows are the same prompt, effects are hypothetical, and the value scores are weak; no candidate brief has been run.

A figure from its own output

Figure produced by research/swe-prompt: type_one_error — swe-prompt/experiments/type_one_error.svg, written by the project's own experiment code.

Paper, code and results

More in The software-engineering loop

Which track should the next run work on, and does a smarter pick beat a uniform one?Which track should the next run work on? Four allocation policies (uniform, greedy, bootstrap Thompson, knapsack-UCB) replayed offline on 7 tracks' scored projects: greedy and Thompson reach about 80% of the best-arm oracle against 60% for the current uniform pick, but value per line falls with order in 4 of 7 tracks, so the stationary replay overstates the gain. Stylised, offline only, no live trial; track 10 milestone 4, partial.
How much of the track-allocation gain is luck of the project history?How much of the track-allocation gain is luck of the history? Block bootstrap of the sequential replay: at 20,000 lines greedy and Thompson beat uniform in 98–100% of 200 histories (intervals exclude zero), knapsack-UCB's does not; the small-budget loss does not reproduce. Stylised, offline only; follow-up to swe-decay.
Do the gains from smarter track allocation survive when tracks run out of projects?Do track-allocation gains survive when tracks run out of projects? Sequential, no-replacement replay: at 20,000 lines greedy and Thompson keep most of their lead over uniform (pass A 134 and 130 against 114), but at 5,000 lines they lose to it (24 against 30). Stylised, offline only; track 10 follow-up to swe-allocate.
Do the portfolio's committed results reproduce when the code is re-run?Do committed results reproduce? A seeded sample of 40 projects with a runnable experiment was re-run: 36 (90%, 95% Wilson 77% to 96%) matched their committed results.txt, 3 timed out under load and 1 needed a missing voice model. Preliminary first pass of track 10, milestone 2; it checks reproduction only, not citations. Second sample of 60 (92 distinct projects pooled): 81 reproduced (88%), no finished run differed; re-running the 5 timeouts serially reproduced all, giving 86 of 92 (93%).
When research merges collide cleanly, is the result still right?When research merges collide cleanly, is the result still right? None of 92 clean replays was broken (28.7% of pairs within 60 min conflict); found and removed two duplicate README bullets the build missed.
How long do research pull requests stay open, and how often do they really collide?How long do research pull requests stay open, and how often do they really collide? Median 0.9 min open (95% merged within 30 min); replayed from real creation times, 2 of 58 shared-file PRs (3.4%) conflict.

All 43 The software-engineering loop projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.