How long can a digital twin be wrong before you can prove it, and what does waiting cost?

research/twin-audit

Anytime-valid e-process audit of a digital twin in closed loop: peeking-safe (2.4% vs 58% false alarms), delay set by the controller's own excitation, regret before detection nearly independent of the gap, dither pays only for gross errors.

7 tests

What it found

A Gaussian-mixture e-process on the closed-loop residual audits a deployed twin continuously: it false-alarms 2.4% of the time against 58% for a z-test checked every step. Detection delay follows ≈ 2[ln(1/α)+½ln(1+τ²nU)]/(Δ²U) with U the controller’s own excitation (simulation 0–18% earlier); expected regret accumulated before detection is nearly independent of the gap (8–10 while per-step regret spans 38×); an expensive-control operator needs 11× longer to catch the same gap; and added probing pays only for gross errors (true gain above 2.25× or below 0.35× the twin’s), with the rule failing at the near-cliff edge. Stylised.

A figure from its own output

Figure produced by research/twin-audit: audit_delay — twin-audit/experiments/audit_delay.svg, written by the project's own experiment code.

Paper, code and results

More in Digital twins & sim-to-real

What does a Gaussian-clutter radar simulator do to a CFAR detector's false-alarm rate?What a Gaussian-clutter radar twin does to a CA-CFAR detector: exact false-alarm law under gamma texture (matched to simulation), 9–30× inflation at design 1e-4, 1.9–3.7 dB threshold repair, shared texture is harmless, and a real-cell calibration budget (median 1.1× at 1000 cells).
Does a more robust CFAR detector survive a Gaussian-clutter radar simulator?Does a more robust CFAR survive a Gaussian-clutter twin? CA, OS and log-t all inflate 8–44× at design 1e-4 (no detector escapes), a known-scale oracle inflates as much as CA (the test cell's own texture), and CA detects best at equal real Pfa after repair.
What does a light-tailed simulated human do to a collaborative robot's timeout?When a collaborative robot should give up waiting for a human: J'(τ)=S(τ)(w−c·h(τ)) makes exponential/Weibull simulated humans corner-only (never abort), 3.7% regret in the base case rising to 37–81% for heavy tails and cheap aborts, plug-in lognormal fit regret 1/n (delta method within 11%).
What does a too-clean sensor twin do to a mapper's commit error?What a too-clean sensor twin does to a mapper's commit error: exact real error via the Lundberg exponent (false-hit 0.05 twin vs 0.15 real turns 1e-3 into 1.45e-2, 14.5×), tempering restores the target at 1.4–16× latency, and the exponent needs several hundred real samples to estimate.
Does cloning an expert from logs recorded in a digital twin recover the expert?Cloning an expert from twin-logged demonstrations under actuator saturation: applied-action logs bias the clone even in a perfect twin (gain 0.58 vs 0.96, +9.8% cost), Stein closed form, quieter or looser twins help, narrow coverage fixes it, DAgger with applied labels is worse.
What does a too-clean digital twin do to a student cloned from a privileged teacher?Cloning a privileged teacher from a digital twin into a noisy-sensor student: BC gain gP/(P+s) is set by the twin's noise-to-sensor ratio, a too-clean twin (¼ the noise) gives 0.36–1.11 excess cost vs 0.005–0.10, on-policy imitation is within 3.5% of optimal, cancellation with actuator error is accidental, and matching one real scalar by demonstration noise fixes it in two rounds.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.