Which 95% upper bound on a failure rate actually covers 95% at simulator-rollout sample sizes?
research/free-intervalcov
Which 95% upper bound on a failure rate really covers 95% at rollout sample sizes: exact coverage shows only Clopper-Pearson never undercovers (min 0.950), Wilson dips to 0.93, Jeffreys to 0.85 and Wald to 0.06–0.75; zero-failure certification of p ≤ 1% takes 299 rollouts (CP), 268 (Wilson) or 191 (Jeffreys).
What it found
Exact coverage (no sampling) of six one-sided 95% upper bounds for k failures in n = 30 to 1000 rollouts, true rates 0.2–20%. Only Clopper–Pearson never undercovers (minimum 0.950); Agresti–Coull dips to 0.944, Wilson to 0.926, Jeffreys to 0.85, Wald to 0.06–0.75, rule-of-three plus Wald to 0.81. Certifying p ≤ 1% from zero failures takes 299 rollouts (Clopper–Pearson), 268 (Wilson), 191 (Jeffreys); at n = 100 four of the methods give identical accept sets for a 5% threshold. Assumes iid rollouts; says nothing about the sim-to-real gap.
A figure from its own output
Paper, code and results
More in Digital twins & sim-to-real
- How many fixed steps per orbit does a twin need before an eccentric orbit stops drifting?How many fixed steps per orbit does a twin need before an eccentric orbit stops drifting? Against the exact Kepler solution RK4 needs 64 to 4096 steps per orbit (e = 0 to 0.9), the symplectic pair 8 to 16 times more, and at e = 0.9 every fixed step fails.
- How long does a digital twin of a chaotic system stay accurate? Each e-fold of smaller state error buys 1.12 time units on Lorenz-63 (1/lambda = 1.11)How long does a digital twin of a chaotic system stay accurate? On Lorenz-63 each e-fold of smaller state error buys 1.12 time units (1/lambda = 1.11); a parameter error buys 0.85. Mean over 40 starts, one tolerance.
- How many missed samples can a PD loop tuned on a fixed-period twin survive?How many missed samples can a PD loop tuned on a fixed-period twin survive? With samples lost independently and the last command held, a double-integrator loop with twin poles at 0.5 is mean-square stable only up to a 29% loss rate but almost surely stable up to 61%; a deadbeat twin tolerates 7% in mean square.
- Does an inverted-pendulum loop tuned on a fixed-period twin survive missed samples, and does an interval-aware gain fix it?On a linearised pendulum (L = 0.5 m) a fixed twin-tuned gain survives any miss probability with slow poles unless the plant is very unstable per sample (limit q = 0.21 at ωh₀ = 1.4), a deadbeat tuning fails by q = 0.03 to 0.10, and a gain recomputed from the actual interval was stable at every q in all 15 cases. Linear, noise-free, simulated.
- How many solver iterations does a physics twin need to hold a stack of boxes still?How many solver iterations does a physics twin need to hold a stack of boxes still? A projected Gauss-Seidel solver at 60 Hz needs 16 for 4 boxes, about 128 for 6 and over 2048 for 16 or more, and extra iterations do not reliably help.
- If the twin is optimistic, how often does a rollout test certify a policy that is not safe?Exact: a twin that fails 10% less often than reality lifts the chance of certifying a 5%-failure policy from 0.049 to 0.087 (fixed test, 1% vs 5%), 20% less often to 0.147, and guarding against a 1.5x optimism costs 2.2 times the rollouts
All 164 Digital twins & sim-to-real projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.