How long can a sequential rollout test run, and what does capping it cost?
research/free-sprtcap
How long can a sequential rollout test run, and what does capping it cost? Exact: the SPRT's 99th percentile is 309 rollouts against a 79 mean (1% vs 5%; fixed test 181), 28% of runs exceed the fixed size at the midpoint rate; capping at the fixed size adds 34–38% to the error, a cap of 1.5 times it keeps the error at 0.053 to 0.057 for a mean within 2% of uncapped.
What it found
Exact dynamic-programming study (no sampling) of the stopping-time distribution of Wald’s sequential test for certifying a failure rate at 5% error, and of adding a hard rollout cap that decides by the sign of the likelihood ratio. For 1% vs 5% (fixed test: 181 rollouts) the average is 79 but the 99th percentile is 309, and 28% of runs at the midpoint rate exceed the fixed size. Capping at the fixed size inflates the unacceptable-rate error to about 0.068; a cap of 1.5 times the fixed size brings it to 0.053–0.057 with the mean within 2% of the uncapped test. Four rate pairs, one truncation rule, iid rollouts matching reality; says nothing about the sim-to-real gap.
A figure from its own output
Paper, code and results
More in Digital twins & sim-to-real
- With noisy scores, does a "beat the record by a margin" rule pick the best candidate more often than the best cutoff?With noisy scores, a rule that must beat the earlier best by a margin raises P(pick the true best) from 0.233 to 0.244 at noise 0.5 and 0.143 to 0.149 at noise 1 (n = 50, fresh-trial test); no gain at noise 2; simulation.
- Does a sampled-data loop tuned on a fixed-period twin survive random sampling intervals?Does a sampled-data loop tuned on a fixed-period twin survive random sampling intervals? Uniform jitter up to ±100% never destabilises it, but missed samples do: almost-surely stable up to miss probability 0.556, mean-square stable only to 0.270 (0.05 for a deadbeat tuning); at 0.4 all 2000 simulated runs converge. Stylised scalar model.
- Does retuning the gain on the sampling-interval law, instead of the fixed-period twin, remove the missed-sample penalty?Does retuning the gain on the sampling-interval law, instead of the fixed-period twin, remove the missed-sample penalty? Yes for this plant: the mean-square-optimal gain is stable at every miss probability (worst E g² 0.478 vs a twin-gain limit of 0.270), at the price of a slower nominal loop; a gain designed for q = 0.3 stays stable at every true q. Stylised scalar model.
- What does a 1:N model of a cargo UAV tell you about the 1:1 aircraft?Research programme (proposal stage): per-quantity error budget for a Froude-scaled cargo aircraft model, with a stdlib scaling calculator; first-order result: at 1:10 the model needs 17 to 35% more power than Froude scaling predicts
- Does a physics-shaped twin of a real buck converter beat plain ARX? Only partly: all models sit at 0.5 to 0.9 V free-run error against a 0.06 V quantisation floor, and the winner flips with which half of the 20 ms record is fittedDoes a physics-shaped twin of a real buck converter beat plain ARX? Only partly: all models sit at 0.5 to 0.9 V free-run error against a 0.06 V quantisation floor, and the winner flips with which half of the 20 ms record is fitted. One 20 ms record; natural frequency fitted, no component values.
- How much feedback delay can a digital twin's controller tolerate, and does a fixed-step simulator get that margin right?Exact: a fixed-step twin under-reports a PD loop's delay margin by 18.6% at dt = 0.1 s and 1.0% at 5 ms, because the largest stable delay is the largest n with (n + 1/2) dt below the true margin (25 of 25 cases)
All 164 Digital twins & sim-to-real projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.