How safe is a robot's goal inference when its commit threshold was tuned against a simulated human?

research/intent-twin

Goal inference tuned against a simulated human: exact real error 1/(1+r^h) of a commit threshold, 3.9×–165× over target for a too-rational twin, 2–4.5× slower for a too-irrational one, population floor P(p<½), and calibration from n real cues misses the target 14–42% of the time.

6 tests

What it found

A robot commits to one of two goals when the evidence lead reaches h (a sequential test on binary motion cues); against a human of cue accuracy p the wrong-goal probability is exactly 1/(1+r^h), r=p/(1−p), matched to simulation. A threshold set for error 10⁻³ in a twin with p_s=0.9 gives real error 3.9× / 33× / 165× the target at p=0.8 / 0.7 / 0.6, while a too-irrational twin (p_s=0.6) is safe but 2.0–4.5× slower. A heterogeneous population (sd 0.08–0.10) doubles the error of the mean-human threshold, and humans with p<½ put a floor under any threshold. Calibrating p_s from n real cues misses the target with probability 0.38 / 0.42 / 0.33 / 0.30 / 0.14 at n=10 / 20 / 50 / 200 / 1000; a lower-confidence-bound design cuts that to ≤4% at the cost of time. Stylised: two goals, i.i.d. binary cues, simulated humans.

A figure from its own output

Figure produced by research/intent-twin: real_error — intent-twin/experiments/real_error.svg, written by the project's own experiment code.

Paper, code and results

More in Digital twins & sim-to-real

How much does a twin's perfectly synchronised sensor clock over-promise a fused estimate?Sensor clock offset and jitter a twin does not simulate: exact added variance 2σ²[1−(1+s²/ℓ²)^{−½}e^{−δ²/2(ℓ²+s²)}], a synchronised-clock fusion design is 1+ag/(b(a+b))× its claim (+44% at 0.3ℓ) and worse than one sensor beyond 0.225ℓ, better delayed sensors hurt more, clock randomisation fixes it.
What does a twin get wrong when its gain is fitted from a noisy logged input, and does more data fix it?A twin gain fitted from a noisy logged input is attenuated by exactly the reliability ratio λ (0.800 vs 0.802 simulated), more data makes the twin's own 95% interval worse (covers the true gain 53%→0% for n=20→200) with no residual warning, a feedforward controller overshoots by exactly 1/λ, and known-noise or instrument repairs cost variance and fail at small n.
What does a twin with no actuator deadband promise a proportional loop, and what does the real one do?An actuator deadband the twin lacks: real loop rests at exactly d/k while the twin goes to 0, tolerance ε needs k ≥ d/ε, a fitted linear gain is 2Q(d/s) (excitation-dependent, small-signal gain 0), and bisection compensation leaves exact (d−d̂)/k or a (d̂−d)/(2−k) chatter (8 probes: 0.784 → 0.0005).
Does measurement noise, or dither, hide an actuator deadband the twin lacks?Follow-up to deadband-twin: measurement noise flips the linear twin's error from 27× optimistic (s=0.01) to conservative (0.37× at s=0.05), dither helps only without noise, and mild under-compensation beats the exact width under noise (single-seed simulations).
How large a manoeuvre can a twin without an actuator rate limit be trusted for?A twin without an actuator rate limit is exact only for x₀ ≤ r/k; beyond it the real loop overshoots (0.90 of x₀ at x₀/r=10⁴) and settles in O(amplitude) steps, with an empirical overshoot threshold 2n(2n−1) for k=1/n.
Does a twin with a rigid drivetrain miss the hunting cycle caused by gear backlash?A rigid-drivetrain twin says the PI loop converges but gear backlash makes the real loop hunt at 1.3–2.9× the gap (exactly ∝ h); describing-function prediction 7–10% high, and bistable starts can hide it.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.