Does a kernel MMD test catch the same wrong digital twins as a classifier two-sample test?

What it found

A kernel two-sample test (unbiased RBF MMD, median-heuristic bandwidth) is compared with a gradient-boosted classifier two-sample test on the stylised pendulum of mimic-twinc2st. Rejection uses the 95th percentile of the exact twin’s own statistic. Raw-trajectory MMD has essentially no power against a Coulomb-friction error at n = 100–400 per class (0.05–0.06 at the median bandwidth, 0.37 at 0.03× for −20% friction), where the classifier reaches 1.00 at −20% from n = 200. MMD on second differences detects a +25% noise-level error at 0.97 from n = 100, where the raw-feature classifier is at 0.08–0.25. Stylised; 24 classifier replicates per cell, so those intervals are wide.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in Adversarial human imitation

• Does a classifier two-sample test see the integrator, timestep and friction gaps of a MuJoCo digital twin?Does a classifier two-sample test see the integrator, timestep and friction gaps of a MuJoCo digital twin? Stylised pendulum: AUROC 0.98 for Euler at 0.02 s, 0.53 (null) for a paired error of 0.003 rad, 0.92 for Gaussian-vs-Student-t noise at zero paired error. Preliminary, simulated only.
• Does a kernel MMD test catch the same wrong MuJoCo digital twins as a classifier two-sample test?Does a kernel MMD test catch the same wrong MuJoCo digital twins as a classifier two-sample test? Stylised MuJoCo pendulum: boosted-tree C2ST power 1.00 at n = 200 for Euler at 0.02 s and a noise-shape error where MMD stays near 0.05; MMD on second differences catches a +25% noise-level error; friction errors undetected by all at n <= 200.
• Does a digital twin that a classifier can tell apart from the real system also make a worse controller?Does a digital twin that a classifier can tell apart from the real system also make a worse controller? Stylised pendulum: 6 of 15 wrong twins have zero controller regret at C2ST AUROC 0.58 to 0.999; the worst regret (3.8%) comes from a twin with no sensor noise and zero paired error. Preliminary, simulated only.
• On a swing-up task where friction matters, does a digital twin's detectability predict what tuning on it costs?On a swing-up task where friction matters, does a twin's detectability predict what tuning on it costs? Stylised pendulum: only the two gravity-error twins cost much (3.7% and 8.3% extra real cost); 13 others cost under 0.3% at C2ST AUROC 0.51 to 1.0. Preliminary, simulated only.
• Does a finer controller grid change which wrong digital twins cost something on a swing-up task?Does a finer controller grid change which wrong pendulum twins cost something on the swing-up task? No: the same two gravity-error twins cost 3.8% and 6.1%, the rest under 0.3%; detectability vs regret ρ = 0.38 (p = 0.16). Preliminary, simulated only.
• Does a wider gain grid, or a controller that uses the twin's gravity, change which wrong digital twins are costly?Does a wider gain grid, or a controller that uses the twin's gravity, change which wrong pendulum twins are costly? Gravity errors stay the costly ones; giving the controller the twin's g/L made g/L −20% regret 13 times larger (0.013 to 0.177). Preliminary, simulated only.

All 41 Adversarial human imitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.