Does a permutation test on the cross-validated statistic keep its size and beat the holdout rule's power on a digital twin?

What it found

Follow-up to free-c2stexact on the stylised pendulum twin. A permutation test on the pooled cross-validated AUROC (39 label permutations, p ≤ 0.05) falsely rejects 0.040 of the time at n = 100 and 0.020 at 200 (100 replicates), while for a 10% Coulomb error at n = 200 it rejects 0.367 against 0.183 for the half-and-half holdout rule (60 replicates, intervals just overlapping). The unadjusted cross-validated rule rejects 0.600 but is not a valid test. Costs about 40 times the training. A first run using p < 0.05 on the k/40 grid had an effective level of 0.025 and was rerun. Stylised, simulated, one classifier, few replicates.

A figure from its own output

Paper, code and results

• README and method
• Experiments and results
• Tests

More in Digital twins & sim-to-real

• What does a quadratic thrust curve do to a controller designed in a linear-thrust twin?A controller that inverts a linear-thrust twin meets a quadratic thrust curve: realised acceleration is a+a²/(4g), so climbs overshoot more (0.046→0.095) and descents less, and steps below p≈0.5 hardly notice.
• Does inverting a quadratic thrust curve exactly fix the error a linear-thrust twin makes, and what if the thrust exponent is also wrong?Exact square-root thrust inverse vs the linear one on a quadratic-thrust vehicle: up/down overshoot gap 3.00→0.09 points with the right model, but only 3.67→1.65 when the thrust exponent is misidentified (2.5 vs 2).
• How much does a wrong process-noise guess cost a Kalman filter?How much does a wrong process-noise guess cost a Kalman filter? Exact scalar steady state: under-guessing q by ×100 costs ×7.4 MSE, over-guessing by ×100 only ×1.6 (bounded by r), and the under-guessing filter reports 5× too-small error bars. Stylised.
• Do low-discrepancy scenarios make a twin evaluate a policy more accurately than iid domain randomisation?Do low-discrepancy scenarios beat iid domain randomisation for evaluating a policy in a twin? Latin hypercubes only cut MSE 1.6–12× (rate stays n^-0.5); shifted Halton changes the rate for smooth returns (149× lower MSE at d=2) and keeps 4–24× on a success-rate jump, shrinking with d; the R_d lattice is erratic at d=5; 95% intervals from 8 randomisations cover 94–97%. Stylised.
• Why does the R_d low-discrepancy lattice lose its advantage at d = 5 when evaluating a twin policy?Why does the R_d low-discrepancy lattice lose its edge at d=5 when scoring a twin policy? Not a seed fluke (exponent −0.45 ± 0.01 over six seeds) and not a smooth dimension effect (a dip between d=4 and d=6); a √p generator gets −0.66 at d=5. A simple resonance score does not explain it. Stylised.
• How long does a float32 twin of a chaotic system shadow the float64 one, and do its statistics survive?How long does a float32 twin of a chaotic system shadow the float64 one, and do its statistics survive? Lorenz and Rössler: float32 departs from float64 after a median 12.3 and 164 time units, like a float64 start error of about 5×10⁻⁶ (100× the float32 roundoff), but the Lyapunov exponent and time-averaged statistics agree within sampling error (stylised).

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.