What does a too-narrow scenario generator do to a digital twin's estimate of a rare failure?
research/scenario-twin
A too-narrow scenario generator in a twin makes rare-failure estimates silent: reweighted variance finite iff s²>½, 0.6× width claims 2e-7 for a real 1.2e-3 and reports zero failures in all runs, optimal width ≈1.05·a needs 38× fewer runs than plain Monte Carlo, and a defensive real-prior mixture is finite but no better.
What it found
With real scenarios x~N(0,1), failure iff x>a, and a twin generator of width s, the raw twin claims Q(a/s) and the reweighted (importance-sampling) estimate has per-sample second moment sτQ(a/τ), τ2=s2/(2s2−1), finite only for s2>½. In a braking example (P=1.2×10−3) a generator 0.6× too narrow claims 2.1×10−7 and reported zero failures in all 4,000 simulated runs of 2,000 scenarios; a wide one over-claims but reweighting repairs it, with optimal width ≈1.05a needing 2,179 runs for 10% error against 83,459 for plain Monte Carlo. A defensive mixture with the real prior makes a narrow twin's variance finite but no better than plain Monte Carlo. Stylised: one-dimensional Gaussian scenarios, known threshold and density, correct twin failure indicator.
A figure from its own output
Paper, code and results
More in Digital twins & sim-to-real
- What does a state estimator tuned in a twin with the wrong noise levels cost in the real world?An estimator tuned in a twin with wrong noise levels: exact real MSE of the twin's Kalman gain, optimistic claim when the twin is too quiet (+121% at 10×) vs pessimistic when too noisy, no instability cliff, and a Mehra innovation retune with regret ∝ 1/n.
- What does a controller designed in a twin with the wrong actuator gain cost in the real world?A controller designed in a twin with the wrong actuator gain: exact real LQR cost, an instability band the twin does not warn about, closed-loop non-identifiability without dither, and a dithered retune with matched regret law.
- How long does a simulator without actuator limits say an unstable plant stays in control?A limit-free twin of an unstable plant with a saturating actuator overstates mean time to loss of control by 6,295× at noise sd 1 (1.5×10⁶ vs 245 steps), an exponential-in-1/s² gap (slope ≈3.7 real vs ≈12.7–13.2 twin), though both rank the deadbeat gain best.
- What does a Kalman filter tuned in an ideal-sensor twin cost on a quantizing real sensor?A Kalman filter tuned in an ideal-sensor twin, run on a quantizing sensor: twin gain costs +11% (
R=1,D=4) to +64% (R=0.01), SheppardR+D²/12retune is within 1% of best, but its MSE claim under-predicts by up to 43% once the step exceeds the state's scale. - How should a simulation budget be split across twin fidelities?Splitting a simulation budget across twin fidelities (multilevel Monte Carlo): smooth quantity saves 3.6×–897× (rates α=1, β=2), a failure probability only β=1 so multilevel loses at loose targets (0.4×) and reaches 7.7× at 1e-4; equal samples per level cost 156×/3.9× the optimum.
- How many real steps does a twin-pretrained policy save, and which twin errors does fine-tuning inherit?Fine-tuning a twin-pretrained policy on the real plant: exact SGD risk, a twin start saves 64–67% of real steps at 0.6× the cold error but is negative transfer beyond 1×, and twin error decays only as
b²/(4eηk)(flat directions inherited: 13 vs 1,541 steps at equal initial loss).
All 164 Digital twins & sim-to-real projects →
Other research topics
Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1
Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.