Can a digital twin shorten a real-world safety certification?

research/twin-certification

Can a twin shorten a real-world safety certification? Worst-case size of any k ≤ c rule is exactly P(Bin(n,ε) ≤ c), so a twin prior cannot cut real trials at a fixed level (299 exact zero-failure trials; twin prior reaches 133 only at size 0.26); wrong twins inflate size or block certification.

11 tests

What it found

Certifying a failure rate p < ε from n real trials with a twin-derived prior. The worst-case false-certification probability of any ‘certify iff k ≤ c’ rule is exactly P(Bin(n,ε) ≤ c), so by Karlin–Rubin the twin cannot cut real trials at a fixed level: at ε=0.01, δ=0.05 the exact test needs 299 zero-failure trials, a twin prior reaches 133 only at worst-case size 0.26, and the exact test held to the same size needs the same n (1057 vs 1057 for 90% power at p=ε/3). A weakly informative Jeffreys bound is already anti-conservative (size 0.147 at 191 trials). Optimistic twins inflate size, pessimistic ones (claim 2ε) block certification entirely, and the largest safe twin weight is about 90 twin-trial equivalents at n=299. Stylised: one Bernoulli failure rate, no continuous margins.

A figure from its own output

Figure produced by research/twin-certification: wrong_twin — twin-certification/experiments/wrong_twin.svg, written by the project's own experiment code.

Paper, code and results

More in Digital twins & sim-to-real

How optimistic is the cost a planner quotes from its own twin rollouts?The cost a planner quotes from its own twin rollouts is optimistic by exactly e_K·s (winner's curse; 95% intervals cover 60% at K=20), fresh rollouts fix the sampling part at a selection-quality price, but not the twin's own model error −e_K τ²/√(τ²+s²).
How many retries does a grasp need when the simulator treats attempts as independent?How many retries a grasp needs when the twin treats attempts as independent: exact Beta-mixture failure law (power law k^{−a} vs geometric), twin budget 6 claims 0.004 and delivers 0.010–0.151, real budget 7–571, closed-form give-up rule, and a two-attempts-per-object repair that still needs a margin.
How optimistic is the best of K controllers chosen in a noisy digital twin?The best of K controllers picked in a noisy twin: winner's claim optimistic by exactly τ²c_K/√(v²+τ²) (c_K=E max of K normals), under a fixed rollout budget more candidates can make the deployed one worse (best K=5–22), an exact-plant LQ twin's claim falls below what any gain can achieve, and screen-then-verify cuts claim bias 4.5×.
How many real rollouts does a biased digital twin save when evaluating a robot policy?Evaluating a robot policy with a biased twin as a control variate: unbiased whatever the twin's bias (+0.700 twin-only vs +0.001), exact variance σ²((1−ρ²)/n+ρ²/N), ρ=η for a faithful twin but √η for a noise-free one (1.33× vs 2.0× real-sample gain), worth-it threshold c_T/c_R<(1−√(1−ρ²))²/ρ² (0.072 at ρ=0.5), and 86–125 vs 164 real episodes per policy to rank correctly when twin bias favours the wrong one.
How big a pilot decides whether a biased digital twin is worth using?How big a pilot decides whether a biased twin is worth using: threshold ρ*=2√r/(1+r) with a gain linear in ρ−ρ*, so wrong calls are frequent but cheap (regret 0.004–0.016 at ρ=0.5), decision-regret law f·slope·(1−ρ*²)²/(2m) (ratio 0.63→1.00 for m=10→160), plug-in allocation ≈0.27–0.30/m, and an adaptive two-stage design that stays unbiased (|bias| ≤ 0.002) with 0.946–0.955 coverage.
Does giving two policies the same seed in a digital twin actually pair them?Seed-paired (common random numbers) policy comparison in a twin: correlation is exactly lost to random-stream desynchronisation (flat score 0.95→0.77, terminal score 0.36→0.001 at q=0.05→0.3), a sizing trap (power 0.11–0.85 vs nominal 0.90), and counter-keyed draws restore ρ=1.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.