Which paired test of a digital twin survives both heavy-tailed cells and whole bad runs?

research/free-robustcombo

Which paired test of a digital twin survives both heavy-tailed cells and whole bad runs? Stylised pendulum, 10% Coulomb error: a two-level statistic (trimmed mean of squared residuals within each run, then trimmed mean across runs) has power 0.99–1.00 at n = 50 under t(2) cells, bad runs at 0.86 and both at once, where the earlier single-level statistics each fail one case. Costs power under Gaussian noise (0.84 against 0.91).

8 tests

What it found

Follow-up to free-robusttails, which found no single robust statistic won against both Student-t(2) cells and 10%-of-runs-10×-noisier bad runs. Here each trajectory’s squared residuals are summarised robustly (median or 20%-trimmed mean), then the per-run values are summarised robustly across runs. On the same stylised pendulum (10% Coulomb error, n = 50–200, 400 null replicates) the trimmed-then-trimmed statistic has power 0.99–1.00 under t(2) cells and when t(2) cells sit inside bad runs, 0.86 / 0.95 / 1.00 under bad runs alone, and 0.84 / 0.99 / 1.00 under Gaussian noise, against 0.91 / 0.99 / 1.00 for the per-run mean-squared median; the best single-level statistics reach only 0.10–0.28 or 0.15–0.38 in the case they fail. Size is 0.02–0.08 against a nominal 0.05. One toy pendulum, one twin error, one contamination design; stylised and simulated only.

A figure from its own output

Figure produced by research/free-robustcombo: power_vs_n — free-robustcombo/experiments/power_vs_n.svg, written by the project's own experiment code.

Paper, code and results

More in Digital twins & sim-to-real

Which significance rule gives a classifier two-sample test the right false-rejection rate on a digital twin?Which significance rule gives a classifier two-sample test the right false-rejection rate on a digital twin? Stylised pendulum: cross-validated rules falsely reject 0.06 to 0.13 at a nominal 0.05, a holdout rank-sum test holds 0.04 to 0.06 at roughly twice the data cost. Preliminary, simulated only.
Does a permutation test on the cross-validated statistic keep its size and beat the holdout rule's power on a digital twin?Does a permutation test on the cross-validated statistic keep its size and beat the holdout rule's power on a digital twin? Stylised pendulum: null rates 0.040 and 0.020, power 0.367 vs 0.183 for holdout at n=200 (Coulomb -10%).
What does a quadratic thrust curve do to a controller designed in a linear-thrust twin?A controller that inverts a linear-thrust twin meets a quadratic thrust curve: realised acceleration is a+a²/(4g), so climbs overshoot more (0.046→0.095) and descents less, and steps below p≈0.5 hardly notice.
Does inverting a quadratic thrust curve exactly fix the error a linear-thrust twin makes, and what if the thrust exponent is also wrong?Exact square-root thrust inverse vs the linear one on a quadratic-thrust vehicle: up/down overshoot gap 3.00→0.09 points with the right model, but only 3.67→1.65 when the thrust exponent is misidentified (2.5 vs 2).
How much does a wrong process-noise guess cost a Kalman filter?How much does a wrong process-noise guess cost a Kalman filter? Exact scalar steady state: under-guessing q by ×100 costs ×7.4 MSE, over-guessing by ×100 only ×1.6 (bounded by r), and the under-guessing filter reports 5× too-small error bars. Stylised.
Do low-discrepancy scenarios make a twin evaluate a policy more accurately than iid domain randomisation?Do low-discrepancy scenarios beat iid domain randomisation for evaluating a policy in a twin? Latin hypercubes only cut MSE 1.6–12× (rate stays n^-0.5); shifted Halton changes the rate for smooth returns (149× lower MSE at d=2) and keeps 4–24× on a success-rate jump, shrinking with d; the R_d lattice is erratic at d=5; 95% intervals from 8 randomisations cover 94–97%. Stylised.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.