How do you estimate a one-in-a-million failure from a twin you can only run a thousand times?

research/tilt-twin

Importance sampling of a rare twin failure: exact mean-shift variance, 10⁸→535 runs for 10% error at p=1e-6, a narrow shift target (2× the optimum is 1.7·10⁵× worse, 0% coverage) and effective sample size as the only warning.

6 tests

What it found

Shifting the twin's disturbances and reweighting failures is unbiased for any shift, with an exact variance e^nθ²Q((b+nθ)/√n) (5.53 vs 5.51 simulated). At the optimal shift, 10% error at p=10⁻⁶ takes 535 runs instead of 10⁸. But the shift is a narrow target: half the optimum costs 23× the variance, twice the optimum 1.7×10⁵×, and a shift tuned for one design is 1.6–17.6× worse at thresholds 0.8–1.5× away. The usual 95% interval covers 94.9% at the optimum but 2.8% at twice it, and only the effective sample size warns. Stylised: Gaussian additive disturbances, constant shift, no field data.

A figure from its own output

Figure produced by research/tilt-twin: design_drift — tilt-twin/experiments/design_drift.svg, written by the project's own experiment code.

Paper, code and results

More in Digital twins & sim-to-real

Which ego-velocity estimator should a radar twin choose when the real scene has moving targets?Radar ego-velocity from Doppler returns: a static-world twin picks least squares, but with moving targets its variance is exactly 1+ετ² times the claim (95% interval covers 48.7% at ε=0.1, τ=10) and the median wins on a band of ε (0.006–0.97 at τ=10; never at τ=1).
A radar twin with no Doppler folding certifies an ego-velocity estimator at every speed: what does the folding sensor do to it, and when can it be unfolded?A radar twin with no Doppler folding: the twin-tuned ego-speed estimator breaks at v_u−2σ with an exact closed-form bias (saturating at one fold off), and a wrapped-cost search unfolds it iff azimuths are diverse, with a confusion probability Φ(−√E₁/2σ) matching Monte Carlo.
How many real trials before “the twin passed validation” means anything for a rare failure?Validating a twin's rare-failure rate against real trials: an exact difference test "passes" a 3×-wrong twin 81% of the time at 500 trials, rejecting a 2× error needs ~11 real failures (any q), and an equivalence test needs ~20/58/178 failures for margins 2/1.5/1.25 (~20/q trials).
How safe is a robot's goal inference when its commit threshold was tuned against a simulated human?Goal inference tuned against a simulated human: exact real error 1/(1+r^h) of a commit threshold, 3.9×–165× over target for a too-rational twin, 2–4.5× slower for a too-irrational one, population floor P(p<½), and calibration from n real cues misses the target 14–42% of the time.
How much does a twin's perfectly synchronised sensor clock over-promise a fused estimate?Sensor clock offset and jitter a twin does not simulate: exact added variance 2σ²[1−(1+s²/ℓ²)^{−½}e^{−δ²/2(ℓ²+s²)}], a synchronised-clock fusion design is 1+ag/(b(a+b))× its claim (+44% at 0.3ℓ) and worse than one sensor beyond 0.225ℓ, better delayed sensors hurt more, clock randomisation fixes it.
What does a twin get wrong when its gain is fitted from a noisy logged input, and does more data fix it?A twin gain fitted from a noisy logged input is attenuated by exactly the reliability ratio λ (0.800 vs 0.802 simulated), more data makes the twin's own 95% interval worse (covers the true gain 53%→0% for n=20→200) with no residual warning, a feedforward controller overshoots by exactly 1/λ, and known-noise or instrument repairs cost variance and fail at small n.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.