Does a classifier two-sample test see the integrator, timestep and friction gaps of a MuJoCo digital twin?

What it found

A MuJoCo hinge pendulum (RK4, 0.002 s, dry friction, Student-t sensor noise) is the ‘real’ system; twins change the integrator, timestep or friction. A boosted-tree classifier two-sample test (800 trajectories per class, 3 seeds) has AUROC 0.98 for semi-implicit Euler at 0.02 s (paired RMSE 0.028 rad), 0.90 at 0.01 s, 0.67 at 0.005 s (0.007 rad) and 0.53 at 0.002 s (0.003 rad), which is not separable from the null 0.51. A friction error of 10% gives 0.52 and 20% gives 0.61. Gaussian instead of Student-t noise of the same sd has zero paired error but AUROC 0.92. Limits: one toy system, one classifier, 3 seeds; the ‘real’ system is itself simulated. Stylised, preliminary.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in Adversarial human imitation

• Does a kernel MMD test catch the same wrong MuJoCo digital twins as a classifier two-sample test?Does a kernel MMD test catch the same wrong MuJoCo digital twins as a classifier two-sample test? Stylised MuJoCo pendulum: boosted-tree C2ST power 1.00 at n = 200 for Euler at 0.02 s and a noise-shape error where MMD stays near 0.05; MMD on second differences catches a +25% noise-level error; friction errors undetected by all at n <= 200.
• Does a digital twin that a classifier can tell apart from the real system also make a worse controller?Does a digital twin that a classifier can tell apart from the real system also make a worse controller? Stylised pendulum: 6 of 15 wrong twins have zero controller regret at C2ST AUROC 0.58 to 0.999; the worst regret (3.8%) comes from a twin with no sensor noise and zero paired error. Preliminary, simulated only.
• On a swing-up task where friction matters, does a digital twin's detectability predict what tuning on it costs?On a swing-up task where friction matters, does a twin's detectability predict what tuning on it costs? Stylised pendulum: only the two gravity-error twins cost much (3.7% and 8.3% extra real cost); 13 others cost under 0.3% at C2ST AUROC 0.51 to 1.0. Preliminary, simulated only.
• Does a finer controller grid change which wrong digital twins cost something on a swing-up task?Does a finer controller grid change which wrong pendulum twins cost something on the swing-up task? No: the same two gravity-error twins cost 3.8% and 6.1%, the rest under 0.3%; detectability vs regret ρ = 0.38 (p = 0.16). Preliminary, simulated only.
• Does a wider gain grid, or a controller that uses the twin's gravity, change which wrong digital twins are costly?Does a wider gain grid, or a controller that uses the twin's gravity, change which wrong pendulum twins are costly? Gravity errors stay the costly ones; giving the controller the twin's g/L made g/L −20% regret 13 times larger (0.013 to 0.177). Preliminary, simulated only.
• Does domain randomization over a wrong digital twin cut the controller regret of tuning on one wrong twin?Does tuning a controller on a spread of randomized twins cut the regret of tuning on one wrong twin? Stylised pendulum: it brings the no-noise twin's regret from 0.0098 to 0.00006, but a very wide range raises a doubled-noise twin's worst-seed regret 7.5-fold. Preliminary, no LLM.

All 41 Adversarial human imitation projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.