What is the cheapest capped certification test in the worst case, not just at the two design rates, and what does it cost at the ends?

What it found

free-sprtopt found the test with the smallest average of the mean rollouts at p₀ and p₁ and listed the minimax design as its next step. Both are linear programmes in the occupation measure of the test (the number of surviving paths at each node (n, f) of the rollout lattice): every error probability and every mean is linear in it, so HiGHS returns the exact optimum over all randomised tests, with no sampling. The average LP reproduces the dual lower bound of free-sprtopt to two decimals (98.78 for 1% vs 5% at a cap of 181), which validates the construction. The minimax LP controls the mean on a grid of failure rates and adds the worst rate found by an exact evaluation until the grid and the exact maximum agree to 0.001. Across three (p₀, p₁) pairs and caps of 1–3× the fixed size, the worst-case mean falls 0.6–0.8% at a cap equal to the fixed size and 5.5–10% once the cap is 2× or more, because the average-optimal test only controls its two ends and its mean in between rises with the cap (133.4, 131.0 then 134.8 for 1% vs 5% at 1×, 2× and 3×) while the minimax value saturates (123.8 from 1.5×). The worst case sits at a rate between the two (1.9% for 1% vs 5%). Versus the fixed test the minimax design needs 32% fewer rollouts at worst for 1% vs 5% (123.8 against 181), and its mean at the design rates is 3–6% higher than the average-optimal test (69.9 and 110.8 against 66 and 107 at p₀ and p₁). Rollouts are assumed iid Bernoulli with the twin’s failure rate (no sim-to-real gap); the worst case is taken over p in [p₁, p₀] only, and the exact maximum is checked on a 161-point grid, not proved over the continuum; nodes the controlled rates reach with probability under 10⁻⁹ are dropped from the LP and treated as reject; every number is exact, nothing is sampled.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in Digital twins & sim-to-real

• How few rollouts can a capped certification test need while keeping both errors at 5%, and how far is the tuned SPRT from that optimum?Follow-up to free-sprtcut: backward induction gives the optimal capped test, 98.9 rollouts on average for 1% vs 5% at a cap of 181 (proved within 0.2% by a dual bound), against 107.3 for the grid-tuned SPRT and 181 for the fixed test; the tuned SPRT is 3 to 9% above the optimum at every cap tried.
• Is the overflow twin's lead over black-box models on a real two-tank rig statistically real? A 0.13 V RMSE gap over ARX 4/4 with a block-bootstrap 95% interval of 0.06 to 0.20 VIs the overflow twin's lead over black-box models on a real two-tank rig statistically real? A 0.13 V RMSE gap over ARX 4/4 with a block-bootstrap 95% interval of 0.06 to 0.20 V; the iid bootstrap is about three times too narrow. Test-trace sampling only, not fit variance.
• Does the overflow twin's lead over ARX survive refitting on a real two-tank rig? A 0.13 V gap in 100% of refits with 87.5% of the training data, but 0.07 V and 84 to 90% of refits with 50 to 75%Does the overflow twin's lead over ARX survive refitting on a real two-tank rig? A 0.13 V gap in 100% of refits with 87.5% of the training data, but 0.07 V and 84 to 90% of refits with 50 to 75%. Overlapping windows, fixed optimiser start.
• Does a physics digital twin beat black-box models on a real two-tank rig? 0.53 V test RMSE with 8 parameters vs 0.66 V for the best black boxDoes a physics digital twin beat black-box models on a real two-tank rig? An 8-parameter overflow twin gets 0.53 V test RMSE against 0.66 V for the best black box (9-parameter ARX); quadratic NARX models diverge in free run. One rig, one split, no confidence interval.
• Can a 22-DOF home robot be printed in PLA around one $15 servo, and which of its twin's simplifications matter for learning to reach and pick?A 3D-printable 22-DOF bimanual mobile manipulator (two 7-DOF arms with geared elbows and parallel grippers, head, lift, omni base) on one $15 servo, $627–948 in parts, with a MuJoCo twin generated from the same parameters as the CAD: reach and pick policies trained on the exact twin ignore servo, friction and mass errors but fail at one 40 ms step of latency (reach 99→29%, pick 91→41%); randomisation fixes reaching almost for free but costs picking 14 points; an interference sweep over the printed solids caught an elbow range the twin allowed but the parts could not reach. Simulated only, not yet built.
• How few real trials certify a robot policy when a biased simulator supplies the rest?Research programme (milestones 3 to 6 logged): valid, bias-agnostic sim-to-real evaluation of robot policies, consolidating about 130 twin projects into one question, with 13 checked references, hypotheses H1 to H6 and CoRL as the target venue

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.