If the twin is optimistic, how often does a rollout test certify a policy that is not safe?
research/free-sprtbias
Exact: a twin that fails 10% less often than reality lifts the chance of certifying a 5%-failure policy from 0.049 to 0.087 (fixed test, 1% vs 5%), 20% less often to 0.147, and guarding against a 1.5x optimism costs 2.2 times the rollouts
What it found
Exact dynamic-programming study (no sampling) of what a constant twin-to-real optimism ratio does to a 5%-error rollout certificate, for the fixed binomial test and Wald’s sequential test. For 1% vs 5% (fixed test: 181 rollouts) the chance of certifying a policy whose real failure rate is 5% rises from 0.049 to 0.087 when the twin fails 10% less often, 0.147 at 20% less and 0.526 at 50% less; the sequential test is within 0.02 of it, so neither is more robust. It doubles to 10% at a ratio of 0.87 (0.93 for the narrower 5% vs 10% and 10% vs 20% pairs, where a 30% optimistic twin certifies more often than not). Guarding against optimism up to R by testing p0/R costs 1.27 times the rollouts at R = 1.1, 2.17 at 1.5 and 4.55 at 2 for 1% vs 5%, and cannot be done for the narrow pairs at R = 2. Four rate pairs, iid rollouts, a pure ratio bias that is an input, not measured from any real twin.
A figure from its own output
Paper, code and results
More in Digital twins & sim-to-real
- Can a rollout cap on a sequential certification test be set without losing the 5% error guarantee, and what does it cost?Follow-up to free-sprtcap: re-tuning a capped SPRT's boundaries and final cut so both errors stay at or below 5% exactly costs, for 1% vs 5% (fixed test 181 rollouts), a mean of 107 rollouts at a cap equal to the fixed size and 90 at twice it, against 90 to 95 for the plain capped test that overshoots the error; the fixed test needs 181.
- What is the cheapest capped certification test in the worst case, not just at the two design rates, and what does it cost at the ends?Follow-up to free-sprtopt: an exact linear programme gives the capped certification test with the smallest worst-case mean over failure rates between p1 and p0; for 1% vs 5% at a cap of 362 it needs 123.8 rollouts at worst against 131.0 for the average-optimal test and 181 for the fixed test, at the price of 3 to 6% more rollouts at the two design rates.
- How few rollouts can a capped certification test need while keeping both errors at 5%, and how far is the tuned SPRT from that optimum?Follow-up to free-sprtcut: backward induction gives the optimal capped test, 98.9 rollouts on average for 1% vs 5% at a cap of 181 (proved within 0.2% by a dual bound), against 107.3 for the grid-tuned SPRT and 181 for the fixed test; the tuned SPRT is 3 to 9% above the optimum at every cap tried.
- Is the overflow twin's lead over black-box models on a real two-tank rig statistically real? A 0.13 V RMSE gap over ARX 4/4 with a block-bootstrap 95% interval of 0.06 to 0.20 VIs the overflow twin's lead over black-box models on a real two-tank rig statistically real? A 0.13 V RMSE gap over ARX 4/4 with a block-bootstrap 95% interval of 0.06 to 0.20 V; the iid bootstrap is about three times too narrow. Test-trace sampling only, not fit variance.
- Does the overflow twin's lead over ARX survive refitting on a real two-tank rig? A 0.13 V gap in 100% of refits with 87.5% of the training data, but 0.07 V and 84 to 90% of refits with 50 to 75%Does the overflow twin's lead over ARX survive refitting on a real two-tank rig? A 0.13 V gap in 100% of refits with 87.5% of the training data, but 0.07 V and 84 to 90% of refits with 50 to 75%. Overlapping windows, fixed optimiser start.
- Does a physics digital twin beat black-box models on a real two-tank rig? 0.53 V test RMSE with 8 parameters vs 0.66 V for the best black boxDoes a physics digital twin beat black-box models on a real two-tank rig? An 8-parameter overflow twin gets 0.53 V test RMSE against 0.66 V for the best black box (9-parameter ARX); quadratic NARX models diverge in free run. One rig, one split, no confidence interval.
All 164 Digital twins & sim-to-real projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.