How optimistic is the cost a planner quotes from its own twin rollouts?

What it found

A planner picks the cheapest of K plans by n twin rollouts each and explains the choice by quoting that simulated cost. With equal true costs the quote is low by exactly e_K · s (e_K = expected maximum of K standard normals: 0.56 at K=2, 1.16 at 5, 1.87 at 20; Monte-Carlo within 0.5%), a nominal 95% interval around it covers the true cost 88%, 78% and 60% of the time at K=5, 10, 20, and two identical plans are quoted a gap of 1.13 s. Reporting from fresh rollouts removes the bias exactly but costs selection quality (regret 0.069 to 0.104, best-plan rate 0.76 to 0.68 when 8 of 20 rollouts are diverted). It does not remove the twin’s own model error b_k ~ N(0,τ²): the quoted-minus-real bias is −e_Kτ²/√(τ²+s²) (formula −0.531, simulated −0.530 at τ=0.5), so only real trials of the chosen plan give an honest real-cost quote. Stylised: independent Gaussian sample means (one skewed check: formula 7% high), no human study, no robot.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Digital twins & sim-to-real

• How many retries does a grasp need when the simulator treats attempts as independent?How many retries a grasp needs when the twin treats attempts as independent: exact Beta-mixture failure law (power law k^{−a} vs geometric), twin budget 6 claims 0.004 and delivers 0.010–0.151, real budget 7–571, closed-form give-up rule, and a two-attempts-per-object repair that still needs a margin.
• How optimistic is the best of K controllers chosen in a noisy digital twin?The best of K controllers picked in a noisy twin: winner's claim optimistic by exactly τ²c_K/√(v²+τ²) (c_K=E max of K normals), under a fixed rollout budget more candidates can make the deployed one worse (best K=5–22), an exact-plant LQ twin's claim falls below what any gain can achieve, and screen-then-verify cuts claim bias 4.5×.
• How many real rollouts does a biased digital twin save when evaluating a robot policy?Evaluating a robot policy with a biased twin as a control variate: unbiased whatever the twin's bias (+0.700 twin-only vs +0.001), exact variance σ²((1−ρ²)/n+ρ²/N), ρ=η for a faithful twin but √η for a noise-free one (1.33× vs 2.0× real-sample gain), worth-it threshold c_T/c_R<(1−√(1−ρ²))²/ρ² (0.072 at ρ=0.5), and 86–125 vs 164 real episodes per policy to rank correctly when twin bias favours the wrong one.
• How big a pilot decides whether a biased digital twin is worth using?How big a pilot decides whether a biased twin is worth using: threshold ρ*=2√r/(1+r) with a gain linear in ρ−ρ*, so wrong calls are frequent but cheap (regret 0.004–0.016 at ρ=0.5), decision-regret law f·slope·(1−ρ*²)²/(2m) (ratio 0.63→1.00 for m=10→160), plug-in allocation ≈0.27–0.30/m, and an adaptive two-stage design that stays unbiased (|bias| ≤ 0.002) with 0.946–0.955 coverage.
• Does giving two policies the same seed in a digital twin actually pair them?Seed-paired (common random numbers) policy comparison in a twin: correlation is exactly lost to random-stream desynchronisation (flat score 0.95→0.77, terminal score 0.36→0.001 at q=0.05→0.3), a sizing trap (power 0.11–0.85 vs nominal 0.90), and counter-keyed draws restore ρ=1.
• What overshoot does a controller tuned in a fixed-step digital twin really have?What overshoot a controller tuned in a fixed-step twin really has: exact per-step decay laws for explicit/semi-implicit/implicit Euler, damping bias flips sign at ζ=1/√2 (explicit, implicit), a 10% twin target delivers 0.105–0.192 real overshoot (implicit/semi-implicit) or 0.036–0.091 (explicit), error first order in h, Richardson repair.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.