Does a classifier two-sample test catch digital-twin errors that a paired trajectory error misses?

What it found

A stylised pendulum with Coulomb friction and heavy-tailed (Student-t) sensor noise is the ‘real’ system; twins differ in friction, noise shape or noise level. A classifier two-sample test (5-fold CV AUROC, 800 vs 800 trajectories, 3 seeds) separates real from twin trajectories with boosted trees: a friction error of 0.03 (10%) that moves the noise-free trajectory by only 0.006 rad RMSE — a quarter of the 0.02 rad noise sd — gives AUROC 0.72, and 0.91 at 20%. A twin with the right dynamics but Gaussian instead of Student-t noise of the same sd has paired error exactly 0 yet AUROC 0.92 once second-difference features are added (0.59 on raw angles). A logistic regression on raw angles stays at chance (0.51) throughout, because the start angles are symmetric. Limits: one toy system, two noise models, three seeds, a handful of classifier settings; simulated only, nothing about real hardware. Preliminary, no LLM involved: a side piece of track 11.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in Adversarial human imitation

• Does a kernel MMD test catch the same wrong digital twins as a classifier two-sample test?Does a kernel MMD test catch the same wrong digital twins as a classifier two-sample test? Stylised pendulum: raw-trajectory MMD is blind to a Coulomb-friction error (power 0.05 to 0.06) that boosted trees catch from n = 200, but MMD on second differences detects a +25% noise-level error at 0.97 from n = 100.
• Does a classifier two-sample test see the integrator, timestep and friction gaps of a MuJoCo digital twin?Does a classifier two-sample test see the integrator, timestep and friction gaps of a MuJoCo digital twin? Stylised pendulum: AUROC 0.98 for Euler at 0.02 s, 0.53 (null) for a paired error of 0.003 rad, 0.92 for Gaussian-vs-Student-t noise at zero paired error. Preliminary, simulated only.
• Does a kernel MMD test catch the same wrong MuJoCo digital twins as a classifier two-sample test?Does a kernel MMD test catch the same wrong MuJoCo digital twins as a classifier two-sample test? Stylised MuJoCo pendulum: boosted-tree C2ST power 1.00 at n = 200 for Euler at 0.02 s and a noise-shape error where MMD stays near 0.05; MMD on second differences catches a +25% noise-level error; friction errors undetected by all at n <= 200.
• Does a digital twin that a classifier can tell apart from the real system also make a worse controller?Does a digital twin that a classifier can tell apart from the real system also make a worse controller? Stylised pendulum: 6 of 15 wrong twins have zero controller regret at C2ST AUROC 0.58 to 0.999; the worst regret (3.8%) comes from a twin with no sensor noise and zero paired error. Preliminary, simulated only.
• On a swing-up task where friction matters, does a digital twin's detectability predict what tuning on it costs?On a swing-up task where friction matters, does a twin's detectability predict what tuning on it costs? Stylised pendulum: only the two gravity-error twins cost much (3.7% and 8.3% extra real cost); 13 others cost under 0.3% at C2ST AUROC 0.51 to 1.0. Preliminary, simulated only.
• Does a finer controller grid change which wrong digital twins cost something on a swing-up task?Does a finer controller grid change which wrong pendulum twins cost something on the swing-up task? No: the same two gravity-error twins cost 3.8% and 6.1%, the rest under 0.3%; detectability vs regret ρ = 0.38 (p = 0.16). Preliminary, simulated only.

All 41 Adversarial human imitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.