Can a twin identified from logs of the deployed controller tell you whether a different controller is stable?

research/closedloop-twin

A twin identified from logs of the deployed controller: held-out error is 1.00σ² at every dither level while the predicted pole under a new gain has sd (σ/√n)√((k−k')²/τ²+1/v) (0.037→0.82), no dither gives a confident wrong twin (b̂=−0.22 vs +1), point-rule false stability certification 3–48% vs 0.1–4.7% for the OLS bound.

8 tests

What it found

A twin fitted to closed-loop logs predicts the pole under a new gain k′ with sd (σ/√n)√((k−k′)²/τ²+1/v), set by the dither τ: held-out one-step error is 1.00–1.005σ² at every τ, while the predicted pole's sd goes 0.037 to 0.82 (matching the formula to a few percent). With no dither the minimum-norm fit gets b̂=−0.22 (true +1) and certifies a real-unstable gain in 100% of fits. Point-rule false certification is 3–48%; the OLS upper bound cuts it to 0.1–4.7% but rarely certifies a good gain at low dither (5–9%). A predicted-pole sd of 0.05 needs τ=1.23 / 0.40 / 0.16 at n=200 / 1000 / 5000. Stylised: scalar linear plant, simulated data.

A figure from its own output

Figure produced by research/closedloop-twin: false_cert — closedloop-twin/experiments/false_cert.svg, written by the project's own experiment code.

Paper, code and results

More in Digital twins & sim-to-real

How much of a digital twin's error is that it was fitted to finite real data, however long it is simulated?A twin fitted to finite real data: exact delta-method error √(1+(2−ρ)²)/((1−ρ)√n) of its steady-state wait, data needed grows as (1−ρ)⁻² (22 100 per stream for 10% at ρ=0.9), exact instability probability, false SLA certification 36–46% fixed by bootstrap/posterior bounds (delta bound under-covers).
For a fixed budget, how much should a fitted digital twin spend on real data and how much on simulation length?Real data or more simulation for a fitted twin: relative variance a(ρ)/n + b(ρ)/m (matches simulation), one real pair ≈ 2 simulated jobs at any load, optimal real-data share 0.57–0.97 by cost ratio, and a simulation-only interval whose real coverage falls 0.79→0.17 as the run lengthens while the input-aware one stays 0.92–0.95.
When should a twin run that counts rare failures stop?When to stop a twin run counting rare failures: sequential Wald stopping halts on zero failures (90% of runs at p=10⁻³, coverage 9.6%), inverse sampling to m failures gives exactly unbiased (m−1)/(N−1) with relative sd ≈1/√(m−2) and an exact interval (95.0–95.5%), while a fixed run planned for the wrong p delivers 63–200% half-width.
What stopping distance can a twin with independent terrain cells certify?Stopping distance certified by a twin with independent per-cell friction: exact incomplete-gamma law (matches braking simulation), 100 twin cells certify 1.17× the mean distance but real terrain with 1–10 patches exceeds it 44%–18% (claimed 0.1%), finer cells are worse, the mean is nearly right, and the measured patch count repairs it.
What approach speed can a lidar twin that returns every beam certify?Approach speed certified by a lidar twin with independent beam dropout vs real whole-scan fades: exact scan-product miss law, real miss ~100× the level at the twin's speed, speed off 10–30% (probabilities off by orders of magnitude), sign reverses when several returns are needed, repaired by a few hundred calibration scans.
How do you estimate a one-in-a-million failure from a twin you can only run a thousand times?Importance sampling of a rare twin failure: exact mean-shift variance, 10⁸→535 runs for 10% error at p=1e-6, a narrow shift target (2× the optimum is 1.7·10⁵× worse, 0% coverage) and effective sample size as the only warning.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.