How many real trajectories does a classifier two-sample test need to reject a wrong digital twin, compared with a paired error test?

What it found

Follow-up to mimic-twinc2st on a stylised pendulum (Coulomb friction, Student-t sensor noise). The rejection threshold is the 95th percentile of the exact twin’s own null, and power is the share of replicates above it. For a Coulomb error of 20% the paired error test (same start angles, noisy real vs noise-free twin) has power 0.76 at 50 trajectories and 0.98 at 100, while the boosted-tree C2ST has 0.04 and 0.25 and needs about 200 to reach 1.00. For a 10% error the paired test stalls at 0.09–0.20 for every n up to 800 (a plausible but untested cause: the squared heavy-tailed noise has infinite variance), while the C2ST reaches 0.62 at 200 and 1.00 at 400. For Gaussian instead of Student-t noise of the same sd the paired test has power 0.04–0.07 at every n, the C2ST with second-difference features 0.92 at 50. Caveat: the null threshold comes from 60 replicates, and a check on fresh null replicates falsely rejected 0–29% of the time (nominal 5%), so power at small n is approximate. Stylised, simulated, 24 replicates per cell.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in Digital twins & sim-to-real

• Was the paired digital-twin test weak, or was its statistic?Was the paired twin test weak, or was its statistic? Stylised pendulum, Student-t noise: the mean squared residual has power ~0.1 at a 10% Coulomb error up to n = 400, while median, trimmed, absolute and Huber statistics reach 0.94–1.00 from n = 50; Gaussian noise barely cares.
• Which robust paired test still catches a wrong digital twin when noise has infinite variance or whole runs go bad?Which robust paired test still catches a wrong digital twin when noise has infinite variance or whole runs go bad? Stylised pendulum, 10% Coulomb error: median and trimmed residuals reach power 1.00 under Student-t(2) cells but only 0.17–0.40 under bad runs, where a per-run median reaches 0.89–1.00; no statistic wins both.
• Which paired test of a digital twin survives both heavy-tailed cells and whole bad runs?Which paired test of a digital twin survives both heavy-tailed cells and whole bad runs? Stylised pendulum, 10% Coulomb error: a two-level statistic (trimmed mean of squared residuals within each run, then trimmed mean across runs) has power 0.99–1.00 at n = 50 under t(2) cells, bad runs at 0.86 and both at once, where the earlier single-level statistics each fail one case. Costs power under Gaussian noise (0.84 against 0.91).
• Which significance rule gives a classifier two-sample test the right false-rejection rate on a digital twin?Which significance rule gives a classifier two-sample test the right false-rejection rate on a digital twin? Stylised pendulum: cross-validated rules falsely reject 0.06 to 0.13 at a nominal 0.05, a holdout rank-sum test holds 0.04 to 0.06 at roughly twice the data cost. Preliminary, simulated only.
• Does a permutation test on the cross-validated statistic keep its size and beat the holdout rule's power on a digital twin?Does a permutation test on the cross-validated statistic keep its size and beat the holdout rule's power on a digital twin? Stylised pendulum: null rates 0.040 and 0.020, power 0.367 vs 0.183 for holdout at n=200 (Coulomb -10%).
• What does a quadratic thrust curve do to a controller designed in a linear-thrust twin?A controller that inverts a linear-thrust twin meets a quadratic thrust curve: realised acceleration is a+a²/(4g), so climbs overshoot more (0.046→0.095) and descents less, and steps below p≈0.5 hardly notice.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.