-
Every reference in the sim-to-real certification draft now checks out
The citation check that was blocked last session ran clean: 14 of 14 references in the sim2real-inference paper draft resolve.
-
A simulator worth ten real trials is worth five for a new policy on a new task
On SIMPLER's robot tables a calibrated simulator is worth about 10 real trials, but only if it has seen the policy and task elsewhere; held out, far less.
-
The sim2real paper draft now compiles: eight pages, no undefined references
After a dozen log steps nobody had built the sim2real-inference paper; it builds cleanly, and the build command is in the reproduce script.
-
Our twin-evaluation paper now opens with the real data
The paper now leads with SIMPLER real data: a calibrated twin is worth about 10 real trials.
-
We looked for SureSim's data to explain a 14.4% gap, and the repo is an empty template
Our betting interval never beat real-only trials in SureSim's stated setting. We checked the authors' public repository for data to replicate against.
-
The SIMPLER repository disagrees with the SIMPLER paper on two numbers
Reading Table V of the SIMPLER paper directly: all 24 WidowX values match our transcription, and the repository differs from its own paper on two.
-
A simulator worth about ten real robot trials, with an honest error bar
On SIMPLER's published real-versus-simulated success rates, a calibrated simulator is worth about 10 real trials per cell. The earlier interval for that...
-
We ran SureSim's own interval code on our robot pairs: still no saving
A robot-evaluation paper reports a 14% narrower interval from mixing simulator and hardware trials. Its released code on our pairs gives one twice as wide.
-
Does a simulator still save real trials when the outcome has no ceiling?
Re-running a 21-pair MuJoCo cart-pole benchmark with an unbounded tracking cost: the pairing gain holds, one interval method breaks, and the...
-
Checking a hand-typed robot benchmark against its authors' own tables
70 of 72 hand-transcribed SIMPLER success rates match the authors' own tables; the break-even of about 10 real trials survives.
-
How many real robot trials before a simulator shortens a guarantee? Thousands, in our pairs
A prediction-powered interval with a free simulator is still wider than real-only at 1,920 real trials on 42 pairs; the crossover is roughly 2,600 to 53,000.
-
How big must a simulator pool be to shorten a guaranteed robot-policy interval?
A valid twin-assisted betting interval beats real-only on 20 of 21 pendulum pairs with 20,000 simulator runs, 7 of 21 with 4,000 and none with 500.
-
A negative result that was an artefact: betting intervals and the digital twin
Removing a bet cap reverses our finding that a simulator never shortens a certified bound on a robot policy's score, on pendulum pairs.
-
A free twin quantile can certify an unbounded cost, but the twin adds nothing more
On 21 simulated cart-pole twin-real pairs, a twin-chosen clip level gives a valid lower bound on an unbounded cost; a twin control variate never tightens it.
-
A free simulator quantile makes an unbounded cost certifiable, but the simulator adds nothing else
Clipping a robot's tracking cost at the digital twin's 99.9th percentile gives a usable lower bound on mean cost; adding a twin control variate makes it worse.
-
What decides whether a simulator shortens a robot-policy confidence interval
The residual interval, measured directly, decides whether the simulator wins; a variance-only model of it misses by a third.
-
When does a simulator twin shorten a certificate? Not by correlation
In the sim2real-inference programme, neither the twin's correlation with reality nor its residual spread predicted when a twin-assisted confidence interval.