Can a rollout cap on a sequential certification test be set without losing the 5% error guarantee, and what does it cost?
research/free-sprtcut
Follow-up to free-sprtcap: re-tuning a capped SPRT's boundaries and final cut so both errors stay at or below 5% exactly costs, for 1% vs 5% (fixed test 181 rollouts), a mean of 107 rollouts at a cap equal to the fixed size and 90 at twice it, against 90 to 95 for the plain capped test that overshoots the error; the fixed test needs 181.
What it found
free-sprtcap showed that simply capping Wald's SPRT inflates the unacceptable-rate error (0.067–0.069 at a cap equal to the fixed test's size). This project re-tunes it: an exact dynamic programme over (trials, failures) searches multiples of Wald's two boundaries (0.5 to 2.5×, each side separately) and every possible final cut for the setting that holds both achieved errors at or below 0.05 and minimises the mean rollouts averaged over the acceptable and unacceptable rates. For p1 = 1% against p0 = 5% (fixed test 181 rollouts) the mean is 107 at a cap of 1× the fixed size, 99 at 1.2–1.3×, 94 at 1.5× and 90 at 2×, always well under 181; the plain capped test is cheaper at small caps (90 to 94) only because it breaks the error guarantee, and at 1.75–2× the re-tuned test is cheaper than it (90.3 against 95.0) while holding the guarantee. Two other (p0, p1) pairs show the same pattern. Rollouts are assumed iid Bernoulli with the twin's failure rate (no sim-to-real gap), the boundary grid is coarse (steps of 0.1×), only the average of the two means is minimised, and nothing is sampled: every number is exact.
A figure from its own output
Paper, code and results
More in Digital twins & sim-to-real
- What is the cheapest capped certification test in the worst case, not just at the two design rates, and what does it cost at the ends?Follow-up to free-sprtopt: an exact linear programme gives the capped certification test with the smallest worst-case mean over failure rates between p1 and p0; for 1% vs 5% at a cap of 362 it needs 123.8 rollouts at worst against 131.0 for the average-optimal test and 181 for the fixed test, at the price of 3 to 6% more rollouts at the two design rates.
- How few rollouts can a capped certification test need while keeping both errors at 5%, and how far is the tuned SPRT from that optimum?Follow-up to free-sprtcut: backward induction gives the optimal capped test, 98.9 rollouts on average for 1% vs 5% at a cap of 181 (proved within 0.2% by a dual bound), against 107.3 for the grid-tuned SPRT and 181 for the fixed test; the tuned SPRT is 3 to 9% above the optimum at every cap tried.
- Is the overflow twin's lead over black-box models on a real two-tank rig statistically real? A 0.13 V RMSE gap over ARX 4/4 with a block-bootstrap 95% interval of 0.06 to 0.20 VIs the overflow twin's lead over black-box models on a real two-tank rig statistically real? A 0.13 V RMSE gap over ARX 4/4 with a block-bootstrap 95% interval of 0.06 to 0.20 V; the iid bootstrap is about three times too narrow. Test-trace sampling only, not fit variance.
- Does the overflow twin's lead over ARX survive refitting on a real two-tank rig? A 0.13 V gap in 100% of refits with 87.5% of the training data, but 0.07 V and 84 to 90% of refits with 50 to 75%Does the overflow twin's lead over ARX survive refitting on a real two-tank rig? A 0.13 V gap in 100% of refits with 87.5% of the training data, but 0.07 V and 84 to 90% of refits with 50 to 75%. Overlapping windows, fixed optimiser start.
- Does a physics digital twin beat black-box models on a real two-tank rig? 0.53 V test RMSE with 8 parameters vs 0.66 V for the best black boxDoes a physics digital twin beat black-box models on a real two-tank rig? An 8-parameter overflow twin gets 0.53 V test RMSE against 0.66 V for the best black box (9-parameter ARX); quadratic NARX models diverge in free run. One rig, one split, no confidence interval.
- Can a 22-DOF home robot be printed in PLA around one $15 servo, and which of its twin's simplifications matter for learning to reach and pick?A 3D-printable 22-DOF bimanual mobile manipulator (two 7-DOF arms with geared elbows and parallel grippers, head, lift, omni base) on one $15 servo, $627–948 in parts, with a MuJoCo twin generated from the same parameters as the CAD: reach and pick policies trained on the exact twin ignore servo, friction and mass errors but fail at one 40 ms step of latency (reach 99→29%, pick 91→41%); randomisation fixes reaching almost for free but costs picking 14 points; an interference sweep over the printed solids caught an elbow range the twin allowed but the parts could not reach. Simulated only, not yet built.
All 164 Digital twins & sim-to-real projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.