How many real trials before “the twin passed validation” means anything for a rare failure?

What it found

A twin states a failure probability q; the real rate is c q. An exact two-sided binomial difference test “passes” a twin that is 3× wrong with probability 0.81 at 500 trials (q = 10⁻³), and rejecting a 2× error with 80% power needs about 11 real failures, independent of q (normal approximation 9.9). Passing barely moves the odds that a twin is off by 2× (0.325 prior, 0.31 after passing at 0.3 expected failures). An equivalence test (TOST) caps false validation at 5% but needs about 20, 58 and 178 real failures for margins 2, 1.5 and 1.25, so certifying a rare-failure twin within 2× takes about 20/q real trials. Stylised: one Bernoulli failure, known q, no scenario shift, illustrative prior.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Digital twins & sim-to-real

• How safe is a robot's goal inference when its commit threshold was tuned against a simulated human?Goal inference tuned against a simulated human: exact real error 1/(1+r^h) of a commit threshold, 3.9×–165× over target for a too-rational twin, 2–4.5× slower for a too-irrational one, population floor P(p<½), and calibration from n real cues misses the target 14–42% of the time.
• How much does a twin's perfectly synchronised sensor clock over-promise a fused estimate?Sensor clock offset and jitter a twin does not simulate: exact added variance 2σ²[1−(1+s²/ℓ²)^{−½}e^{−δ²/2(ℓ²+s²)}], a synchronised-clock fusion design is 1+ag/(b(a+b))× its claim (+44% at 0.3ℓ) and worse than one sensor beyond 0.225ℓ, better delayed sensors hurt more, clock randomisation fixes it.
• What does a twin get wrong when its gain is fitted from a noisy logged input, and does more data fix it?A twin gain fitted from a noisy logged input is attenuated by exactly the reliability ratio λ (0.800 vs 0.802 simulated), more data makes the twin's own 95% interval worse (covers the true gain 53%→0% for n=20→200) with no residual warning, a feedforward controller overshoots by exactly 1/λ, and known-noise or instrument repairs cost variance and fail at small n.
• What does a twin with no actuator deadband promise a proportional loop, and what does the real one do?An actuator deadband the twin lacks: real loop rests at exactly d/k while the twin goes to 0, tolerance ε needs k ≥ d/ε, a fitted linear gain is 2Q(d/s) (excitation-dependent, small-signal gain 0), and bisection compensation leaves exact (d−d̂)/k or a (d̂−d)/(2−k) chatter (8 probes: 0.784 → 0.0005).
• Does measurement noise, or dither, hide an actuator deadband the twin lacks?Follow-up to deadband-twin: measurement noise flips the linear twin's error from 27× optimistic (s=0.01) to conservative (0.37× at s=0.05), dither helps only without noise, and mild under-compensation beats the exact width under noise (single-seed simulations).
• How large a manoeuvre can a twin without an actuator rate limit be trusted for?A twin without an actuator rate limit is exact only for x₀ ≤ r/k; beyond it the real loop overshoots (0.90 of x₀ at x₀/r=10⁴) and settles in O(amplitude) steps, with an empirical overshoot threshold 2n(2n−1) for k=1/n.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.