How wrong is a safety margin certified in a Gaussian-disturbance simulator when real gusts are heavy-tailed?

research/safety-twin

A corridor-keeping safety margin certified in a Gaussian-disturbance twin: same variance, heavy-tailed real gusts give 0.29 real failure at a claimed 0.01, needed margin 7.3× at δ=0.001 (ν=3), variance inflation misses by 24×, conformal needs n≥1/δ, a Hill tail fit from one-step data repairs it.

9 tests

What it found

A stabilised vehicle must stay in a corridor for T = 200 steps; the twin’s disturbances are Gaussian, the real ones Student-t with the same variance, so a variance fit cannot tell them apart. The twin’s certificate is exact (a union bound over its stationary tail), yet at the margin certified for δ = 0.01 the real failure probability is 0.29 (ν = 3) to 0.052 (ν = 8), and the margin the real system needs at δ = 0.001 is 7.3× to 1.25× the twin’s, a ratio that grows as δ shrinks. Inflating the twin variance to match real failure at δ = 0.05 still misses δ = 0.001 by 24×. A conformal margin from real episodes is valid but needs n ≥ 1/δ of them; a Hill tail fit to cheap one-step disturbances needs no failures and is below target in 93% of repeats at 2,000 samples. Stylised: 1-D linear system, i.i.d. disturbance, a simulated ‘real’ system, no flight data.

A figure from its own output

Figure produced by research/safety-twin: failure — safety-twin/experiments/failure.svg, written by the project's own experiment code.

Paper, code and results

More in Digital twins & sim-to-real

How wrong is the thinnest obstacle a lidar twin certifies as detectable when real beam dropout is clustered?A lidar twin with i.i.d. beam dropout certifies the thinnest detectable obstacle: real clustered dropout (same rate, lag-1 correlation 0.6) misses 41× more at the certified width and the range claim is 4× too long (exact Markov miss law, p·q^(m−1)); a fitted persistence recovers the median width but needs a margin.
Can a digital twin shorten a real-world safety certification?Can a twin shorten a real-world safety certification? Worst-case size of any k ≤ c rule is exactly P(Bin(n,ε) ≤ c), so a twin prior cannot cut real trials at a fixed level (299 exact zero-failure trials; twin prior reaches 133 only at size 0.26); wrong twins inflate size or block certification.
How optimistic is the cost a planner quotes from its own twin rollouts?The cost a planner quotes from its own twin rollouts is optimistic by exactly e_K·s (winner's curse; 95% intervals cover 60% at K=20), fresh rollouts fix the sampling part at a selection-quality price, but not the twin's own model error −e_K τ²/√(τ²+s²).
How many retries does a grasp need when the simulator treats attempts as independent?How many retries a grasp needs when the twin treats attempts as independent: exact Beta-mixture failure law (power law k^{−a} vs geometric), twin budget 6 claims 0.004 and delivers 0.010–0.151, real budget 7–571, closed-form give-up rule, and a two-attempts-per-object repair that still needs a margin.
How optimistic is the best of K controllers chosen in a noisy digital twin?The best of K controllers picked in a noisy twin: winner's claim optimistic by exactly τ²c_K/√(v²+τ²) (c_K=E max of K normals), under a fixed rollout budget more candidates can make the deployed one worse (best K=5–22), an exact-plant LQ twin's claim falls below what any gain can achieve, and screen-then-verify cuts claim bias 4.5×.
How many real rollouts does a biased digital twin save when evaluating a robot policy?Evaluating a robot policy with a biased twin as a control variate: unbiased whatever the twin's bias (+0.700 twin-only vs +0.001), exact variance σ²((1−ρ²)/n+ρ²/N), ρ=η for a faithful twin but √η for a noise-free one (1.33× vs 2.0× real-sample gain), worth-it threshold c_T/c_R<(1−√(1−ρ²))²/ρ² (0.072 at ρ=0.5), and 86–125 vs 164 real episodes per policy to rank correctly when twin bias favours the wrong one.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.