Which significance rule gives a classifier two-sample test the right false-rejection rate on a digital twin?

What it found

Follow-up to free-c2stpower, whose empirical null was noisy. On the exact pendulum twin (200 replicates per cell, boosted trees, nominal 0.05) a naive rank-sum test on cross-validated scores and a t-test on per-fold AUROCs falsely reject 0.060–0.130 of the time, never below 0.060, while a half-and-half holdout split with an exact-null rank-sum test stays at 0.040–0.060 as the theory requires. The price is power: for a 10% Coulomb error holdout rejects 0.20 at n = 200 and 0.92 at 400, against 0.62 and 1.00 for the over-sized cross-validated rule, so it needs roughly twice the data. A permutation-calibrated cross-validated statistic is untried. Stylised, simulated, one classifier.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in Digital twins & sim-to-real

• Does a permutation test on the cross-validated statistic keep its size and beat the holdout rule's power on a digital twin?Does a permutation test on the cross-validated statistic keep its size and beat the holdout rule's power on a digital twin? Stylised pendulum: null rates 0.040 and 0.020, power 0.367 vs 0.183 for holdout at n=200 (Coulomb -10%).
• What does a quadratic thrust curve do to a controller designed in a linear-thrust twin?A controller that inverts a linear-thrust twin meets a quadratic thrust curve: realised acceleration is a+a²/(4g), so climbs overshoot more (0.046→0.095) and descents less, and steps below p≈0.5 hardly notice.
• Does inverting a quadratic thrust curve exactly fix the error a linear-thrust twin makes, and what if the thrust exponent is also wrong?Exact square-root thrust inverse vs the linear one on a quadratic-thrust vehicle: up/down overshoot gap 3.00→0.09 points with the right model, but only 3.67→1.65 when the thrust exponent is misidentified (2.5 vs 2).
• How much does a wrong process-noise guess cost a Kalman filter?How much does a wrong process-noise guess cost a Kalman filter? Exact scalar steady state: under-guessing q by ×100 costs ×7.4 MSE, over-guessing by ×100 only ×1.6 (bounded by r), and the under-guessing filter reports 5× too-small error bars. Stylised.
• Do low-discrepancy scenarios make a twin evaluate a policy more accurately than iid domain randomisation?Do low-discrepancy scenarios beat iid domain randomisation for evaluating a policy in a twin? Latin hypercubes only cut MSE 1.6–12× (rate stays n^-0.5); shifted Halton changes the rate for smooth returns (149× lower MSE at d=2) and keeps 4–24× on a success-rate jump, shrinking with d; the R_d lattice is erratic at d=5; 95% intervals from 8 randomisations cover 94–97%. Stylised.
• Why does the R_d low-discrepancy lattice lose its advantage at d = 5 when evaluating a twin policy?Why does the R_d low-discrepancy lattice lose its edge at d=5 when scoring a twin policy? Not a seed fluke (exponent −0.45 ± 0.01 over six seeds) and not a smooth dimension effect (a dip between d=4 and d=6); a √p generator gets −0.66 at d=5. A simple resonance score does not explain it. Stylised.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.