How many simulator rollouts does a sequential test save when certifying a failure rate?

What it found

Exact dynamic-programming comparison (no sampling) of Wald’s sequential probability ratio test with the smallest fixed-size binomial test, both at 5% error, for deciding between an acceptable and an unacceptable failure rate. At either rate the sequential test uses about half the rollouts (for 1% vs 5%: 79 and 113 on average against 181). Between the two rates the saving shrinks to 6–17% at its worst point in the four cases tried. Assumes iid rollouts that match reality, so it says nothing about the sim-to-real gap.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in Digital twins & sim-to-real

• How long can a sequential rollout test run, and what does capping it cost?How long can a sequential rollout test run, and what does capping it cost? Exact: the SPRT's 99th percentile is 309 rollouts against a 79 mean (1% vs 5%; fixed test 181), 28% of runs exceed the fixed size at the midpoint rate; capping at the fixed size adds 34–38% to the error, a cap of 1.5 times it keeps the error at 0.053 to 0.057 for a mean within 2% of uncapped.
• With noisy scores, does a "beat the record by a margin" rule pick the best candidate more often than the best cutoff?With noisy scores, a rule that must beat the earlier best by a margin raises P(pick the true best) from 0.233 to 0.244 at noise 0.5 and 0.143 to 0.149 at noise 1 (n = 50, fresh-trial test); no gain at noise 2; simulation.
• Does a sampled-data loop tuned on a fixed-period twin survive random sampling intervals?Does a sampled-data loop tuned on a fixed-period twin survive random sampling intervals? Uniform jitter up to ±100% never destabilises it, but missed samples do: almost-surely stable up to miss probability 0.556, mean-square stable only to 0.270 (0.05 for a deadbeat tuning); at 0.4 all 2000 simulated runs converge. Stylised scalar model.
• Does retuning the gain on the sampling-interval law, instead of the fixed-period twin, remove the missed-sample penalty?Does retuning the gain on the sampling-interval law, instead of the fixed-period twin, remove the missed-sample penalty? Yes for this plant: the mean-square-optimal gain is stable at every miss probability (worst E g² 0.478 vs a twin-gain limit of 0.270), at the price of a slower nominal loop; a gain designed for q = 0.3 stays stable at every true q. Stylised scalar model.
• What does a 1:N model of a cargo UAV tell you about the 1:1 aircraft?Research programme (proposal stage): per-quantity error budget for a Froude-scaled cargo aircraft model, with a stdlib scaling calculator; first-order result: at 1:10 the model needs 17 to 35% more power than Froude scaling predicts
• Does a physics-shaped twin of a real buck converter beat plain ARX? Only partly: all models sit at 0.5 to 0.9 V free-run error against a 0.06 V quantisation floor, and the winner flips with which half of the 20 ms record is fittedDoes a physics-shaped twin of a real buck converter beat plain ARX? Only partly: all models sit at 0.5 to 0.9 V free-run error against a 0.06 V quantisation floor, and the winner flips with which half of the 20 ms record is fitted. One 20 ms record; natural frequency fitted, no component values.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.