How few real trials certify a robot policy when a biased simulator supplies the rest?

2026-10-30 · Milestone 7y: finite-pool certificate moves into the main text (a writing step, no experiment)

• Why: the 10-28 entries left "whether the finite-pool result belongs in the main text" open. The abstract already leads with it, but the main text's certificate section said only that none of five constructions helps, which contradicted the abstract's own caveat until the reader reached the appendix.
• Done: paper/main.tex gains one main-text paragraph (m7h to m7j: 20 of 21 pendulum pairs at n = 960 with N = 20,000 twin runs, 7 to 8 with 4,000, none with 500; coverage at least 98%; cart-pole 7 of 168 cells) and contribution (3) now names the condition that reverses the negative result. No number is new; each is copied from the appendix paragraph and the m7i/m7j entries. The note that the floor limits the 1/m term in real runs, not the twin-mean term, is an interpretation, stated as such.
• Per-episode pairs: re-probed Hugging Face for per-episode real outcomes. IVC-liuyuan/Simpler_env holds a SimplerEnv tarball and two code zips; hongyuw/SimplerEnv-lerobot holds a README only. Neither has real-robot episodes. 2027.corl.org returns 502 through the proxy, so the CoRL 2027 style file is still unavailable.
• Next: unchanged (style file when the call is out; per-episode real outcomes need the SIMPLER authors).

2026-10-28 · Per-episode pairs: the blocker is the data, not the host (a check, not an experiment)

• Why: eleven entries since 4e say per-episode pairs are "blocked on the SIMPLER video dataset host". huggingface.co is reachable from this proxy now (200), so I looked at what the dataset holds.
• Outcome: the dataset named in earlier entries, xuanlinli17/simpler-env-eval-example-videos (MIT, listed through the Hugging Face API), has 8 zip files (octo-base 697 MB, octo-small 56 MB, rt1_cokecan 29 MB, three rt_1 checkpoints 213 to 308 MB, rt_1_x 319 MB, rt_2_x 278 MB) and an empty README (front matter only). Its title and the checkpoint names say these are example videos of simulated SimplerEnv evaluations. I did not download or unzip them (about 2.2 GB), so I cannot say what per-episode labels they carry. A search of the Hub for "simpler" and "simplerenv" lists other policy-rollout sets (e.g. IVC-liuyuan/Simpler_env, hongyuw/SimplerEnv-lerobot), none of which I opened. SimplerEnv's repository returns 403 through the proxy, so the earlier "no episode logs" check there is not repeatable today.
• Reading: per-episode pairs need a real-robot episode outcome next to a simulated outcome for the same episode. This dataset can at most supply the simulated half. I found no source for per-episode real outcomes, so the block is data that I cannot locate, not a proxy 403, and "reachable host" will not unblock it. The proxy changes nothing about the paper's one sentence on this.
• Next: ask the SIMPLER authors whether per-episode real outcomes exist (owner step: contacting them publishes under the portfolio's name); otherwise the remaining items are the CoRL 2027 style file and whether the finite-pool result belongs in the main text.

2026-10-28 · Milestone 2s: the synthesis of the 138 *-twin projects (a count, not an experiment)

• Why: milestone 2 promised "a synthesis table of what the twin projects found about rho" and the first entry left it undone. Planned before running: count, by keyword rule, how many projects disclose as stylised, name a real data source, use a paired estimator and state a rho; spot check 12 random projects by hand.
• Outcome (experiments/m2s_twin_audit/): of 138, 137 stylised, 3 name a real source, 11 use a paired estimator, 20 state an explicit rho (0.03 to 1.0, median of per-project maxima 0.845). The spot check agreed on all 12. A first, looser rule (any "real data" phrase) flagged 15 and was wrong on the cases read, so it was tightened before the numbers were kept.
• Reading: the twin cluster supplies no measurement of rho on a real system; this programme's real-data step is the only one.
• Next: unchanged (CoRL 2027 style file; per-episode pairs).

2026-10-27 · Citation check: all 14 references verified (a check, not an experiment)

• Why: the previous entry could not run cite_check.py (arxiv.org and the DOI host returned 403), and CoRL desk-rejects papers that cite non-existent references (7v).
• Outcome: both hosts are reachable now. python3 research/_shared/cite_check.py research/sim2real-inference/paper/refs.bib reports 14 of 14 ok (1 DOI, 13 arXiv ids), each title and year resolving to the entry in refs.bib. No bib entry changed.
• Not done: the CoRL 2027 style file, and whether the finite-pool result belongs in the main text.

2026-10-27 · Test-suite check: one order-dependent failure fixed (a check, not an experiment)

• Why: before treating the draft as ready, I ran the whole tests/ directory in one pytest process (PYTHONPATH=src).
• Outcome: 32 passed, 1 failed: test_benchmark.py::test_analysis_helpers did import analyse, but four experiment directories each ship an analyse.py, and another test module had already cached a different one in sys.modules. Run alone the test passed, so the fault was name shadowing, not the estimator. The test now loads experiments/m3_benchmark/analyse.py by path. 33 passed in 9 s. No result or claim changed.
• Not done: cite_check.py on paper/refs.bib could not run (arxiv.org and the DOI host return 403 through the egress proxy); the 14 identifiers stay unchecked this session.

2026-10-26 · Milestone 7x: abstract leads with the real data (a writing step, no experiment)

• Why: 7w moved SIMPLER to the first results section but left the abstract opening on the MuJoCo result; 7v found the CoRL call wants real-robot or transfer evidence first.
• Done: paper/main.tex abstract now states the SIMPLER result right after the question, then the MuJoCo benchmark, certificates and floor. Wording moved, no number or claim changed. Also fixed an unmatched parenthesis in that sentence. Rebuilt (pdflatex, bibtex, pdflatex x2): 8 pages, no undefined references.
• Next: rebuild in the CoRL 2027 style when the call is out; decide whether the finite-pool result belongs in the main text.

2026-10-26 · Milestone 7w: restructure the paper for CoRL's framing (a writing step, no experiment)

• Why: 7v found the call wants robotics relevance and real-robot or transfer evidence, and that Limitations counts toward the 8 pages. The draft led with MuJoCo toy pairs and carried a dozen robustness paragraphs in the main text.
• Done: paper/main.tex now has Real data (SIMPLER) as the first results section, the MuJoCo benchmark, certificates table and floor theorem after it, and the second task, unbounded outcome, betting-interval, finite-pool, larger-n/SureSim and active-selection paragraphs in an appendix. Contributions list reordered so SIMPLER is (1). No number or claim was changed. It builds (pdflatex, bibtex, pdflatex x2, no undefined references): 8 pages in the generic article class, of which the main text including Limitations ends on page 4 and the appendix starts on page 5, so the 8-page budget has room for moving some appendix material back once the real style file is used.
• Not done: the CoRL 2027 style file is not out yet; the abstract still leads with the MuJoCo result and should be reordered then.
• Next: when the CoRL 2027 call appears, rebuild in its style (owner approves fetching the template); decide whether the finite-pool result belongs in the main text.

2026-10-26 · Milestone 7v: what does the CoRL call actually require? (a check, not an experiment)

• Why: 7t left "check the CoRL page limit and style file against the call" open because the venue site was unreachable. 2026.corl.org is reachable now; I read "Call for Papers" and "Instruction for Authors" (https://2026.corl.org/contributions/instruction-for-authors).
• Findings (CoRL 2026, the latest call published): 8 pages for the main paper at submission (a 9th is added for camera-ready); acknowledgments, references and appendix do not count; double-blind; a Limitations section is required and counts toward the 8 pages; the call asks for real-robot experiments "or convincing evidence that simulation experiments are transferable to real robots" and lists "Theoretical foundations of robot learning, including ... uncertainty quantification" and "Benchmarks and datasets" as topics; a submission must "focus on a core robotics problem". Generative-AI policy: AI tools do not qualify for authorship, authors disclose AI use at submission, and papers citing non-existent references are desk-rejected (checked with GPTZero), so cite_check.py stays mandatory. The 2026 paper deadline was 28 May 2026, so the next realistic target is the CoRL 2027 call, not yet published.
• Not done: the LaTeX template is a Google Drive link on that page. I downloaded it but did not build against it (the session's permission gate declined running a script that fetches and compiles an external style file), so the draft is still in the generic article class and its 8-page main-text count under corl_2026.sty is unmeasured. Our draft has a Limitations section already (main.tex).
• Reading: the draft's framing needs one change for CoRL: the call wants robotics relevance and real-robot or transfer evidence; our real-data evidence is SIMPLER's published tables, which is the closest we have, and the paper should lead with that rather than with MuJoCo toy pairs. A theory-heavy variant would fit AISTATS better, as the proposal already says.
• Next: when the CoRL 2027 call appears, rebuild in its style (owner can approve fetching the template); reorder the paper so SIMPLER leads; keep Limitations inside the 8 pages.

2026-10-26 · Milestone 7u: SureSim's released interval code on our pairs (planned before running)

• Why: m7s found only a template repo and stopped. The paper's project page (suresim-robot-eval.github.io, now reachable) links a different repository, irom-princeton/rapid-policy-evaluation (MIT, commit e9e6cb5, 2026-06-01), which holds their interval code (src/suresim/intervals/ppi.py: ppi_uniform, gold_only_est, mean_ci_eff_corrected_membership_accelerated). The paired hardware scores are not in it (the example scripts read cluster paths; the only data file is 628 hardware trials of grasp/task success without paired simulation scores). So I cannot replicate their numbers, but I can run their estimator. Reading it, it differs from my m7q implementation in two ways that matter (and has the same range): (1) the bet is sqrt(2 ln(2/alpha) / (n sigma2_{t-1})) with n the total summand count, with no ln(1+t) factor (mine, from Waudby-Smith and Ramdas' PrPl-EB, has it), so its bets are larger; (2) the summand range is [-(1+N/n), 2+N/n], width 3+2N/n = 26.3 here, the same width as mine (shifted by one, so no difference to test); (3) a candidate mean is excluded when the capital's running maximum (not its final value) reaches 1/alpha, with theta 1/2 (MARTINGALE returns max(K+, K-) / 2, in logs).
• Method (experiments/m7u_suresim_code/suresim_code.py): vendor those three functions (MIT, attribution in the header), run the m7q setting (alpha 0.1, n 60, N 700, k = 1 and 5 contexts per object, 100 replications, 21 pairs per task, both pools) with their ppi_uniform and their gold_only_est, and report width ratios against their own real-only interval and against my m7q real-only interval, coverage, and the count of pairs narrower than real-only, overall and for rho in [0.6, 0.85].
• H7u-i: with their code, uniform PPI is still wider than their real-only interval on at least 80% of pairs in the rho band (the gap with SureSim's reported 14.4% is not an interval-implementation effect). Refuted if it is narrower on more than 20%.
• H7u-ii: their uniform-PPI interval is narrower than my m7q uniform-PPI interval at the same pairs (median ratio below 1), because of the larger bets (no ln(1+t) factor). (Correction made before running: the range is the same, so only the bet size and the stopping criterion differ; H7u-ii is therefore about those.)
• H7u-iii: coverage is at least 90% in every cell (their running-max criterion is conservative). Refuted otherwise.
• Prior: i likely (about 75%); ii yes; iii yes.
• Caveats set in advance: model-versus-model pairs, not their hardware scores; their bisection assumes the excluded set is an interval around the mean; np.random.shuffle order is theirs (unseeded, I seed it).

### Outcome (experiments/m7u_suresim_code/: suresim_code.py, results/suresim_code.csv, results/summary.txt; 200 replications, 21 pairs per task, analysis as pre-registered)

• Their ppi_uniform interval is wider than their own gold_only_est on every one of 84 pair-by-k cells (median ratio 2.00 to 2.19 per task and k; minimum 1.83), including all 33 pairs with rho in [0.6, 0.85]. Coverage at least 95.5% in every cell.
• H7u-i holds (narrower on 0 of 33 band pairs, against a threshold of at most 20%). H7u-iii holds (min coverage 95.5%). H7u-ii is refuted: their uniform interval is wider relative to real-only (ratio about 2.0 to 2.2) than my m7q uniform interval was (1.34 to 2.0). My guess that larger bets would make it narrower was wrong: their bet shrinks as 1/sqrt(n_total) with n_total = 760, so the summand's variance saving is not spent. Their real-only interval is also 1.05 to 1.56 times wider than my cap-0.99 betting interval at n = 60, which fits their bet cap c = 0.5 (m7h found a cap of 0.5 binding on 99 to 100% of bets), not tested again here.
• Reading: running SureSim's own estimator code on our model-versus-model pairs gives no saving either, in the setting of their paper. So the gap between their 14.4% (and the project page's "over 20%") and our zero is not an interval-implementation effect. What remains is the data: their paired hardware scores, which the repository does not contain (the example scripts read private cluster paths; the one data file, examples/FrankaDP_1877507_object_generalization/data_scaled_rotated/real_uniform_raw_full.tsv, has 628 hardware trials of grasp and task success with no paired simulation score). It cannot be settled without those scores.
• Caveats: model-versus-model pairs; objects are averages of iid contexts; the repository code is at commit e9e6cb5 (2026-06-01) and may differ from the code behind the paper's tables; I used their functions as copied, with a seeded shuffle.
• Next: put the corrected m7s sentence in the paper (done in this step); ask the authors for the paired scores (owner step: contacting them publishes under the portfolio's name).

2026-10-26 · Milestone 7t: does the paper draft build? (a check, not an experiment)

• Why: paper/main.tex had grown through a dozen log steps and was never compiled; the venue bar says "drafted in the venue's format" and a draft that does not build is not one.
• Outcome: installed texlive-latex-base, -recommended and -science (Ubuntu 24.04 packages), ran pdflatex, bibtex, pdflatex twice on the unmodified source. It builds: 8 pages including references, no LaTeX errors, no undefined citations or references (the only warning is h float changed to ht). The figure e_vs_rho.png resolves from paper/. The build command is now the last line of reproduce.sh.
• Reading: the draft is buildable but still in a generic article class, not CoRL's style file, and the page count of 8 includes references; the CoRL page limit and the style file must be checked against the call for papers before submission (not fetched; owner step to submit).
• Next: swap in the venue style file once the CoRL 2026 or 2027 call is reachable; otherwise the programme's remaining open item is real per-episode pairs (blocked on the SIMPLER video dataset host).

2026-10-26 · Milestone 7s: does SureSim ship data or code we can replicate against? (a check, not an experiment)

• Why: 7q's next step was "a faithful replication would need their released data". arXiv.org and the project site are blocked by the egress proxy (403 CONNECT), so I looked for a code repository instead.
• Outcome: GitHub search finds one candidate, abadithela/suresim (first author's account, "Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators", created 2026-02-04, no stars). Shallow clone at commit 463bb2f09d39 (2026-02-04): 352 KB, and it is the unmodified "Research Project Template" README. data/original, data/processed, figures and outputs hold only .placeholder files; src/ (executor, experiment.py, experiment.jl, utils.py, 486 lines) is a generic experiment scaffold and utils.py is plotting helpers only. No betting, WSR or PPI code, no hardware scores, no simulated scores.
• Reading: there is nothing there to replicate against as of that commit, so the 7q gap (our betting construction never beats real-only in SureSim's stated setting, theirs reports 14.4%) stays unexplained, and the paper says so. This does not show the authors have no data elsewhere (a different repo, or the project page, which I could not fetch).
• Next: re-check the repo and the paper's project page when a host is reachable; until then the SureSim comparison stays at "stated setting reproduced, result not".

2026-10-26 · Milestone 7q: does our betting construction win in SureSim's own setting? (planned before running)

• Why: m7g said we "read only the abstract" of SureSim (arXiv:2510.04354) and could not reproduce its 14.4% narrower interval. I have now read the full text. Its setting differs from ours in four ways: alpha = 0.1 (we used 0.05); n = 60 paired, N = 700 sim-only; each "object" label is an average of partial scores over 5 real initial conditions (and 100 simulated ones), which smooths the outcome; mean paired correlation 0.72. Its estimator is the uniform PPI of eq. (3) with a WSR betting interval (range of the summand scaled by (n+N)/n), plus a 2-stage variant (rectifier at 0.9 alpha, sim mean at 0.1 alpha). Its reported gain at N = 700 is 0.160 vs 0.187 for real-only (14.4%).
• Method (experiments/m7q_suresim_setting/suresim_setting.py): on the m3 pendulum and m7 cart-pole pools, an "object" is the average of k iid contexts drawn with replacement (k = 1 and k = 5; R and T averaged over the same contexts), truth = pool mean of R. n = 60, N = 700, alpha = 0.1, 100 replications, 21 pairs per task. Intervals: real-only betting (our interval, cap 0.99); uniform-PPI betting on Delta_i = (n+N)/n (R_i - T_i) xi_i + T_i rescaled from [-a, a+1]; 2-stage (rectifier at 0.09, twin mean from the N objects at 0.01). Report width ratios to real-only, coverage, and the same restricted to pairs with object-level rho in [0.6, 0.85].
• H7q-i: the 2-stage interval is narrower than real-only on at least half of the pairs with rho in [0.6, 0.85] at k = 5 (SureSim's regime). Refuted otherwise.
• H7q-ii: the uniform-PPI interval is wider than real-only in the median at k = 5 (the range (n+N)/n = 12.7 swamps the variance saving in our implementation), i.e. we do not reproduce SureSim's uniform gain. Refuted if its median ratio is below 1.
• H7q-iii: averaging k = 5 raises rho and lowers the 2-stage/real-only ratio relative to k = 1. Refuted otherwise.
• Prior: i about 50/50 (alpha 0.1 and N = 700 help, but N = 700 is below the 4,000 where m7j started to win); ii yes; iii yes.
• Caveats set in advance: model-versus-model pairs, not SureSim's hardware data; objects are averages of iid contexts, not real objects; I do not know their exact bet rule or the range they give WSR for the uniform summand.

### Outcome (experiments/m7q_suresim_setting/: suresim_setting.py, results/suresim_setting.csv, results/summary.txt; 100 replications, 21 pairs per task)

• A first run had a bug of mine: the uniform-PPI summand listed the 60 real-labelled items first, which breaks the exchangeable order betting needs, and coverage fell to 0% on low-rho pairs. Fixed by permuting the summand; the numbers below are from the fixed run. Coverage is now at least 96% in every cell.
• Width relative to real-only betting (median over pairs; n 60, N 700, alpha 0.1). Pendulum k=1: uniform 1.34, 2-stage 1.47 (narrower on 0 and 1 of 21; the one is gain-3 at rho 0.52, ratio 0.93). Pendulum k=5: uniform 2.01, 2-stage 1.86 (0 of 21). Cart-pole k=1: 1.34 and 1.72; k=5: 1.34 and 1.72 (0 of 21). Restricted to rho in [0.6, 0.85] (SureSim's regime): best 2-stage ratio 1.03 (pendulum k=1) and no pair narrower.
• H7q-i refuted: the 2-stage interval is narrower than real-only on 0 of 9, 0 of 11, 0 of 5 and 0 of 7 pairs in the rho band. H7q-ii is not supported as written: uniform-PPI is wider in the median everywhere (1.34 to 2.14), so we still do not reproduce SureSim's 14.4% gain, but "range swamps variance" is my guess, not a tested cause. H7q-iii refuted: averaging 5 iid contexts leaves median rho at 0.83 (pendulum) and 0.85 (cart-pole) and makes the ratio worse on the pendulum (1.47 to 1.86), because the real-only interval shrinks with the smoother labels and the twin-mean term does not.
• Reading: alpha = 0.1, n = 60 and N = 700 are not what separates our result from SureSim's. Our k-averaging is not their smoothing (they average real partial scores over 5 initial conditions and 100 simulated perturbations per object, with matched seeds, for real objects); it does not raise the paired correlation here. The remaining candidates are their score distribution (partial scores of hardware trials), their bet rule or range for the uniform summand, and the 120-object finite bank; I cannot tell which from the paper. Caveats: model-versus-model pairs; grid resolution of the interval search is 1/600 of the rescaled range (about 0.04 in uniform-PPI units), which widens that arm slightly.
• Next: put the corrected SureSim paragraph (full text read, not abstract) in the paper; a faithful replication would need their released data, which the paper's website may hold (not fetched).

2026-10-26 · Milestone 4f: how many real trials per policy does it take to rank policies as well as the twin does? (planned before running)

• Why: m4 to m4e price the twin as a level estimator (break-even n). SIMPLER's own claim is about ranking policies, and a practitioner choosing between two checkpoints cares about order, not level. Ranking has a different noise floor, so the price could differ a lot from n of 10.
• Method (experiments/m4f_ranking/ranking.py): within each of the 8 tasks, every policy pair whose published real rates differ. The twin's pairwise accuracy is the share of those pairs whose simulated order agrees with the real order (ties in sim count 1/2). The real-trial comparison treats the published real rates as the truth and draws m trials per policy from Binomial(m, p); its accuracy is P(correct order) with ties 1/2, computed exactly from the binomial pmfs, for m = 1 to 200. Break-even m_rank = smallest m whose real-trial accuracy reaches the twin's. Interval: 4,000 resamples of tasks (clusters), recomputing both the twin accuracy and m_rank. Repeat on pairs with a real gap of at least 0.15.
• H4f-i: twin pairwise accuracy is at least 0.85 on all pairs. Refuted if below 0.85.
• H4f-ii: m_rank is between 5 and 30 trials per policy (the same order of magnitude as n*). Refuted otherwise.
• H4f-iii: m_rank on gap >= 0.15 pairs is no larger than on all pairs (large-gap pairs are easier for both the twin and the real trials, so I expect the twin's lead to shrink, i.e. m_rank to fall). Refuted if it is larger.
• Prior: i yes (about 0.85 to 0.9); ii yes, around 8 to 15; iii yes.
• Caveats set in advance: the real rates are themselves noisy (n 24 to 75), so treating them as truth flatters both arms and the twin's accuracy is measured against a noisy order; the number is a plug-in, not an inference about the true order. 8 tasks, 6 policies on Google, 3 on WidowX.

### Outcome (experiments/m4f_ranking/: ranking.py, results/summary_4f.txt; analysis as pre-registered)

• 71 within-task policy pairs with a real gap; twin pairwise accuracy 0.887 [0.817, 0.966] (task-cluster bootstrap). Real trials reach that accuracy at m_rank = 11 per policy [5, 38] (0.003 of resamples never reach it within 200). Real-trial accuracy at m = 5, 10, 20, 30, 50, 100: 0.832, 0.884, 0.925, 0.944, 0.962, 0.979.
• Pairs with a real gap of at least 0.15 (52): twin 0.942 [0.875, 0.986], m_rank 10 [5, 27].
• H4f-i holds (0.887 >= 0.85; the lower interval end 0.817 is below it, so it is a point-estimate pass). H4f-ii holds (11 in [5, 30]; the interval reaches 38). H4f-iii holds (10 <= 11) but the difference is one trial and the intervals overlap almost entirely, so it is not a finding.
• Reading: pricing the twin by order rather than by level gives the same answer, about ten real trials per policy, which agrees with n* = 10.1 from m4. Two independent framings landing together makes the order-of-magnitude figure more credible; neither is a precise price (intervals 5 to about 38).
• Caveats: the real rates are noisy themselves (n 24 to 75) and are the "truth" here, so both arms are measured against a noisy order; pairs within a task share policies, so pairs are not independent (hence the task-cluster resampling, which is still coarse with 8 tasks); ties in real rates were dropped (all-zero cells), which removes the easiest pairs for the twin if it also ties them.
• Next: per-episode pairs remain blocked; the paper gets one sentence.

2026-10-26 · Milestone 4e: does the SIMPLER calibration transfer to a held-out task or policy? (planned before running)

• Why: m4 calibrates the twin by leave-one-cell-out, so the 35 training cells include the same task with other policies and the same policy on other tasks. A practitioner wants to certify a new policy or a new task, where those cells are not available. 4c widened the intervals; it did not change the point estimate's leakage.
• Method (experiments/m4e_transfer/transfer.py): same 36 cells and OLS-within-robot calibration as m4; predictions for each cell come from a fit that drops (a) the whole task, (b) the whole policy, (c) both its task and its policy. Report debiased MSE, break-even n = mean p(1-p)/MSE, and Pearson r of prediction with real, against m4's leave-one-cell-out (n 10.1). Policy-level bootstrap intervals (4,000 resamples of policies, as 4c) for each.
• H4e-i: held-out-task calibration has n above m4's 10.1 (the transfer penalty is positive). Refuted if n <= 10.1.
• H4e-ii: held-out-policy n* is below 30 (the twin still saves real trials for a new policy). Refuted otherwise.
• H4e-iii: held-out-task-and-policy calibration is no better than the raw twin (n* undefined or MSE_cal >= MSE_raw). Refuted if calibrated MSE stays below raw.
• Prior: i yes (n* about 15 to 25); ii yes; iii no (calibration still helps, with a smaller margin).
• Not tested: per-episode pairs (the video dataset is not reachable from this proxy; the SimplerEnv repository holds no episode logs, checked 2026-10-26).

### Outcome (experiments/m4e_transfer/: transfer.py, results/summary_4e.txt; analysis as pre-registered, but two hypotheses had a sign error, below)

• Result (debiased MSE of the calibrated twin / break-even n / r with real; policy-resampling 95% interval for n): cell (m4) 0.0138 / 10.1 / 0.908 [3.8, 32.6]; held-out task 0.0136 / 10.3 / 0.910 [4.1, 27.6]; held-out policy 0.0195 / 7.1 / 0.878 [2.4, 27.9]; held-out task and policy 0.0290 / 4.8 / 0.834 [1.9, 15.2]. Raw twin MSE 0.0212 throughout.
• Pre-registration error. n is the number of real trials the twin is worth, so a transfer penalty makes n fall, not rise. H4e-i ("n above 10.1") and H4e-ii ("n below 30") were written with the wrong sign for what I meant. Read literally H4e-i holds trivially (10.3 > 10.1) and H4e-ii holds, but neither tests the intent, so I score the intent: a penalty for a new task and the twin still worth something for a new policy.
• Intent of H4e-i: refuted for tasks. Holding out a whole task changes nothing (MSE 0.0136 against 0.0138): a task-level calibration transfers across these 8 tasks. For policies the penalty is real: n falls from 10.1 to 7.1 (MSE +41%), the prior (n about 15 to 25 under my wrong sign) was moot.
• H4e-ii (intent) holds. For a new policy the twin is still worth about 7 real trials, but the calibrated MSE (0.0195) is only 8% below the raw twin's (0.0212), so most of the m4 calibration gain depended on having seen the policy's other tasks.
• H4e-iii refuted my prior, i.e. holds as stated: with both task and policy held out, calibration is worse than no calibration (MSE 0.0290 against raw 0.0212; n* 4.8). The linear fit on the remaining 7 cells' worth of policies and tasks overfits; the raw twin is the better predictor for a new policy on a new task.
• Reading: the m4 headline ("about ten real trials") is an in-sample-policy figure. For a policy and a task the calibration has not seen, the honest figure is about 5 to 7 trials, and calibration should be dropped for the doubly-new case. Caveats: 36 cells, 7 policies, 8 tasks on two robots (OLS within robot, so WidowX fits use 3 policies); policy-resampling intervals are wide and overlap everywhere, so the ordering of schemes is a point estimate, not a significant difference.
• Next: put the transfer figures into the paper's SIMPLER paragraph; per-episode pairs remain blocked (video dataset host not reachable).

2026-10-26 · Milestone 4c: do the SIMPLER intervals survive resampling tasks and policies, not cells? (analysis written after the first run, not pre-registered)

• Why: m4 and m4b flag their bootstrap as "optimistic (cells cluster)": the 36 cells share 8 tasks and 7 policies, so resampling cells treats correlated rates as independent. The question was how much wider the honest interval is. The expectation I held before running: n* upper end moves from about 27 to somewhere near 40 under two-way resampling, and the conclusion (twin worth about 10 real trials, order of magnitude) holds.
• Method (experiments/m4c_cluster/cluster.py): m4's leave-one-cell-out calibrated predictions kept fixed; resample cells, whole tasks, whole policies, or tasks and policies together (cell kept with multiplicity), 4,000 resamples each, 95% percentile intervals.
• Outcome (results/summary_4c.txt): n point 10.1. Cell [4.9, 27.6]; task [5.3, 24.5]; policy [3.8, 34.1]; two-way [2.7, 53.9], with 8.9% of two-way resamples above n = 30 and 0.4% undefined (zero calibrated MSE). Pearson r 0.90: cell [0.83, 0.95], two-way [0.74, 0.97]. My upper-end guess (about 40) was low; the two-way upper end is 54.
• Reading: clustering widens the n* interval about 2x on the upper side, so the headline should read "about 10 real trials, plausibly 3 to 54", not [5, 27]. The twin still saves real trials in every resampling scheme (lower end above 2). Rank agreement stays high. Caveat: resampling 7 policies or 8 tasks is a coarse approximation of the cluster distribution; degenerate resamples (constant sim or real column) emit numpy warnings and give NaN r, dropped from percentiles.
• Next: per-episode pairs from the SIMPLER video dataset if reachable (paired control variate on real episodes); otherwise the paper's remaining sections.

2026-10-26 · Milestone 4d: which eggplant value is the PDF's? (resolves the open item in 4b; analysis written after the run)

• Why: 4b left two real eggplant rates in dispute (Octo-Base 0.233 vs 0.250, Octo-Small 0.433 vs 0.400) and the m4 n = 30 inference unconfirmed.
• Method (experiments/m4b_crosscheck/pdf_check.py): download the SIMPLER PDF (arXiv:2405.05941), read it with pdftotext -layout, parse Table V, compare the 24 WidowX success values with our transcription and with the authors' REAL_PERF/SIMPLER_PERF.
• Outcome: our transcription matches the PDF on all 24 WidowX values. The authors' repository differs from their own paper on exactly the two disputed values (PDF 0.233 and 0.433; repo 0.250 and 0.400). Appendix B states 24 eggplant trials, but 0.233 = 7/30 and 0.433 = 13/30 are not multiples of 1/24, so the paper's table and its appendix disagree; n = 30 stays an inference (24 and 30 both give n* 10.2 in 4b). The 48 Google Robot values were not re-read against the PDF; they agree with the repository, an independent source.
• Reading: the transcription stands; the disagreement is upstream, and worth stating in the paper as a data caveat. No headline number changes.

2026-10-25 · Milestone 4b: does the SIMPLER transcription match the authors' own tables? (analysis written after the first run, not pre-registered)

• Why: m4's 36 cells were typed in from the PDF, and the log flagged the eggplant n as inferred. The SIMPLER repository ships REAL_PERF and SIMPLER_PERF (simpler_env/utils/metrics.py, MIT), an independent source. I looked at the values before writing the plan, so the outcome below is a check, not a test of a prior.
• Method (experiments/m4b_crosscheck/): crosscheck.py compares all 72 values with a verbatim copy of the two dicts; sensitivity.py re-runs the m4 point estimates (leave-one-cell-out calibrated twin, break-even n*) on the transcription, on the authors' values with eggplant n in {24, 30}, and with the sixth Google task added (place apple in closed top drawer; n_real = 27 because .074, .185, .407 are k/27).
• Outcome: 70 of 72 agree. The two that differ are real eggplant, Octo-Base (ours 0.233, authors' 0.250) and Octo-Small (ours 0.433, authors' 0.400); I cannot tell from here which matches the PDF table (0.250 and 0.400 are both k/20, 0.233 and 0.433 are both k/30), so the n=30 inference in m4 is unconfirmed. Headline numbers do not depend on it: n 10.1 (m4), 10.2 (authors' values, n=30 or 24); adding the sixth task (42 cells) gives n 11.2 to 11.3, Pearson r 0.907, raw debiased MSE 0.019 (from 0.021).
• Reading: the real-data result survives an independent source and a larger cell set. Still policy-level only; per-episode pairs are not published in the repository README or code (not checked in the video dataset).
• Next: the paper "What decides the win" section; if per-episode logs from the SIMPLER video dataset can be reached, test the paired control variate on real episodes.

2026-10-25 · Milestone 7p: paper section for the 7k to 7o chain (writing step, no new experiment)

• Done: the "What decides the win" paragraph in paper/main.tex now covers 7o (flip check, power-scale result 1.32 to 1.03, Spearman 0.62 to 0.51, supported-but-incomplete wording) with the numbers copied from m7o_ceiling/results/summary.txt; paper/reproduce.sh gains the missing commands for 7i, 7k, 7l, 7m, 7o (7m's CSV is an input to 7n).
• Not done: real SIMPLER pairs (milestone 4) where the ceiling question can be asked of real scores; the paper still has one paragraph, not a section, for this chain.

2026-10-25 · Milestone 7o: is the cart-pole loss a near-ceiling artefact? (planned before running)

• Why: 7n found, post hoc, that q = W_R / w_real tracks the mean of R (Spearman 0.81) and guessed that the real-only betting interval benefits from a score near the upper bound. Test it by moving the outcome: (a) flip, R' = 1 - R, T' = 1 - T; (b) power, R' = R^p, T' = T^p with p = ln(1/2)/ln(mean R), so mean R' = 1/2. Same intervals as 7n (alpha 0.05, cap 0.99, f = 0.1; residual at alpha(1 - f), rescaled by 2), n in {240, 960}, 100 replications on a shared subsample, 21 pairs x 2 systems. Code: experiments/m7o_ceiling/ceiling.py.
• H7o-i (symmetry check): the flip leaves q unchanged (max |q_flip - q_raw| <= 0.02), since the hedged interval is symmetric. Refuted otherwise (implementation asymmetry).
• H7o-ii: under the power scale the cart-pole median q falls to at most 1.0 (from about 1.33), i.e. the near-ceiling explanation holds. Refuted if it stays above 1.0.
• H7o-iii: Spearman(q_power, mean R) has |rho| <= 0.3 pooled, against 0.8 raw. Refuted otherwise.
• Prior: H7o-i yes; H7o-ii no (q stays above 1; the cart-pole residual is just wide); H7o-iii partly.
• Not tested: real pairs, other interval constructions. Note the power transform changes the estimand, so widths compare only as the ratio q within a scale.

### Outcome (experiments/m7o_ceiling/: ceiling.py, results/ceiling.csv, results/summary.txt; 84 cells, 100 replications)

• H7o-i holds. The flip changes q by at most 0.000 in any cell: the interval is symmetric, so the flip is a check on the code, not a test of the idea.
• H7o-ii refuted, narrowly. Moving mean R to 1/2 with the power scale takes the cart-pole median q from 1.322 to 1.026 (share of cells with q > 1: 0.93 to 0.52); threshold was 1.0. Pendulum barely moves (0.659 to 0.635; share q > 1 0.05 to 0.00). So about 90% of the cart-pole excess over 1 goes away when the outcome is moved off the ceiling, but the twin is still not a clear win there.
• H7o-iii refuted. Pooled Spearman(q, mean R) falls from 0.615 (this run's raw scale; 0.81 in 7n at f = 0.1 for q/q_var, a different statistic) to 0.509 on the power scale, above the 0.3 cut. Mean R is still informative after the transform, partly because the power scale fixes the mean of R but not the shape of the residual.
• Reading: the near-ceiling explanation accounts for most of cart-pole's disadvantage (q 1.32 to 1.03) but not all; it is now a supported but incomplete mechanism from two systems, model-versus-model pairs, one monotone transform. The estimand changes under the power scale, so this says how the interval behaves, not that the twin would pay on the original scale.
• Next: milestone 7p, put the 7k to 7o chain into the paper draft as one section (finite-twin width: decomposition, ceiling effect, and the negative results on rho), then real SIMPLER pairs (milestone 4), where the ceiling question can be asked of real scores.

2026-10-25 · Milestone 7n: what sets the width of the betting interval for the residual R - T? (planned before running)

• Why: 7m found the twin-mean interval is a small part of the combined width (W_T / w_comb median 0.22 pendulum, 0.056 cart-pole), so the win is decided by the residual interval against the real-only one. That split was inferred from medians; here the residual interval is measured directly. The sd ratio s_res/s_R failed to predict the combined/real ratio at the pair level (7l); the question is whether it fails because of the interval construction (bet cap, range 2 after rescaling, alpha split) rather than because of the variances.
• Method (experiments/m7n_residual_width/residual_width.py): for each system (pendulum, cart-pole), each of the 21 pairs, n in {240, 960}, 200 replications on the same subsample: w_real (betting interval for R at alpha = 0.05, cap 0.99, as 7h/7k) and W_R(f) (betting interval for z = (R - T + 1)/2 at alpha(1 - f), rescaled by 2, f in {0.1, 0.5}). Report q(f) = W_R(f)/w_real per cell, and compare with the variance-only prediction q_var(f) = (s_res/s_R) sqrt(ln(2/(alpha(1-f))) / ln(2/alpha)) (width of a variance-adaptive interval scales as sd times sqrt(log(1/alpha)); this is the model, not a fact about the betting interval).
• H7n-i: q_var predicts q within 20% (median absolute relative error <= 0.2) over the 84 cells per f. Refuted if larger.
• H7n-ii: if H7n-i fails, the miss is systematic in the range: q / q_var correlates (Spearman, |rho| >= 0.5) with the skewness of R (a bounded score near 1 gets a narrower interval than its sd gives). Refuted otherwise.
• H7n-iii: W_R(f) + W_T(N, f) from 7m reproduces the 7j/7k combined widths within 10% (median), which checks the decomposition itself.
• Prior: H7n-i fails (median error 25 to 40%); H7n-ii yes; H7n-iii yes.
• Not tested: real pairs, other systems, other interval constructions.

### Outcome (experiments/m7n_residual_width/: residual_width.py, decomposition.py, results/; 84 cells, 200 replications each, same subsample for both intervals)

• H7n-i refuted. The variance-only prediction q_var misses the measured q = W_R / w_real by a median 32.6% (f = 0.1) and 35.1% (f = 0.5), against a threshold of 20%. Pendulum is close (median error 13.7%, q median 0.66 against q_var 0.57); cart-pole is not (47 to 50%, q median 1.33 against 0.64). On cart-pole the residual interval alone is wider than the whole real-only interval at the median, so no twin-mean cost can make the combined interval win there.
• H7n-ii refuted as stated. Spearman(q / q_var, skewness of R) is -0.23 and -0.24, below the 0.5 threshold (and negative).
• H7n-iii holds. W_R + W_T reproduces the 7j/7k combined width with median relative error 0.4% (90th percentile 1.6%) over 336 cells, and the rule "W_R + W_T < w_real" gives the same win/loss as the measured combined interval in 330 of 336 cells. The decomposition in 7m is an accounting identity, now checked.
• Post hoc, not pre-registered: q / q_var has Spearman 0.81 with the mean of R (0.82 at f = 0.5); the median ratio is 1.07 on the pendulum (mean R 0.67) and 1.90 on cart-pole (mean R 0.84). The reading: the real-only betting interval for a score that sits near the upper bound is narrower than its sd implies (the cap on the bet and the variance of a bounded variable shrink near the edge), while the residual, rescaled to be centred at 1/2, gets no such help. So the twin's disadvantage on cart-pole is partly a property of the interval construction applied to a near-ceiling outcome, not of pair quality. This is a hypothesis from two systems and 84 cells, not a result.
• Next: milestone 7o, test the near-ceiling explanation by shifting the outcome (apply the same interval to 1 - R, and to R after a monotone transform that moves its mean to 1/2) and see whether the real-only advantage and the cart-pole losses move with it; then the paper (this step adds the 7k to 7n paragraph).

2026-10-25 · Milestone 7m: is the finite-twin win a threshold on real-only width over twin-mean width? (planned before running)

• Why: 7l refuted rho and s_res/s_R as predictors and found, post hoc, that the real-only interval width separates wins from losses (97.6% at n = 960), with the caveat that the ratio has w_real in its denominator. The mechanistic claim is: the combined interval is roughly (residual interval) + (twin-mean interval W_T), so the twin pays when w_real exceeds the residual interval plus W_T. Test the threshold form out of sample, with W_T varied directly through N and the alpha share f.
• Method (experiments/m7m_threshold/threshold.py; no new pair simulation): for each system, N and f of the 7j (pendulum, N in {4,000, 20,000}) and 7k (cart-pole, N in {20,000, 100,000}) CSVs, measure W_T as the median width of 40 fresh twin-mean betting intervals (alpha = 0.05 f, iid draws from that system's twin pool, same interval code as 7k). A cell is a (pair, n, N, f); win = combined width < w_real. Statistic x = w_real / W_T. Fit the single cut k on pendulum cells only (maximum accuracy), then score on cart-pole cells with k fixed. Cells per system: 21 pairs x 2 n x 2 N x 2 f = 168.
• H7m-i: the pendulum-fitted cut classifies cart-pole cells win versus not with accuracy >= 90%. Refuted if lower.
• H7m-ii: x predicts the win better than w_real alone in the same out-of-sample test (accuracy of the w_real-only cut fitted on pendulum is lower by >= 5 points), i.e. the twin-mean width carries information. Refuted otherwise.
• H7m-iii: the fitted k is above 1 (twin-mean width has to be a fraction of the real-only width). Refuted if k <= 1.
• Prior: H7m-i partly (80 to 90%); H7m-ii yes because W_T varies by about 5x across cells while w_real does not; H7m-iii yes.
• Not tested: real pairs, other systems, a causal claim.

### Outcome (experiments/m7m_threshold/: threshold.py, results/threshold.csv, results/summary.txt; 168 cells per system, W_T from 40 fresh twin-mean intervals per (system, N, f))

• H7m-i refuted. The cut on x = w_real / W_T fitted on pendulum (3.19; train accuracy 75.0%) classifies 4.2% of cart-pole cells correctly, far below chance: cart-pole x is large (W_T is only 0.0008 to 0.0019) so the rule predicts wins, and cart-pole wins in 7 of 168 cells.
• H7m-ii refuted as stated, and the w_real-only number is not a success. The pendulum-fitted w_real cut scores 94.6% on cart-pole, higher than x, but cart-pole is 95.8% losses, so "never win" scores 95.8%. Train accuracy on pendulum is only 63.7% for w_real alone (75.0% for x), so W_T carries some information within a system and none that transfers.
• H7m-iii holds nominally (k = 3.19 > 1), but the cut it belongs to does not transfer, so it says nothing.
• Post hoc (not pre-registered), from threshold.csv: the twin-mean interval is a small part of the combined width: W_T / w_comb has median 0.22 on pendulum and 0.056 on cart-pole. The rest, w_comb &minus; W_T (the interval for the residual R &minus; T), is 0.70 of w_real at the median on pendulum (range 0.50 to 1.30) and 1.37 on cart-pole (0.88 to 1.70). So whether a twin pays is decided by the residual interval against the real-only interval; the twin-mean cost, and hence N and f, matter at the margin. This corrects the 7l reading ("the real-only width against the fixed twin-mean cost").
• Caveats: two systems, model-versus-model pairs, the residual interval is inferred as a difference of medians, not measured separately; the 7l post hoc width correlation was partly mechanical, as flagged there.
• Next: milestone 7n, measure the residual interval directly and ask what sets its width relative to the real-only one (the sd of R &minus; T over the sd of R was tested in 7l and failed at the pair level, so look at the tails or the bounded-score range the betting interval uses); then the paper draft (milestone 7 still thin).

2026-10-25 · Milestone 7l: what predicts when a twin pays? rho and residual spread against the finite-twin win (planned before running)

• Why: 7j and 7k found wins on 20 of 21 pendulum pairs and at most 2 of 21 cart-pole pairs, and 7k read this as pair quality. That reading is untested. Analysis only, no new simulation: reuse the 7j and 7k result CSVs (ratio = combined width over real-only width, per pair) and recompute per pair, from the m7 score pools, rho and s_res/s_R, the sd of the residual R - T over the sd of R (the quantity the residual-variance argument says matters). Code: experiments/m7l_predictor/predictor.py. Cells: n in {240, 960}, N = 20,000, f = 0.1, both systems (42 pairs per n).
• H7l-i: pooled over both systems, the ratio is monotone in s_res/s_R: Spearman rho <= 0.8 at n = 960. Refuted if lower.
• H7l-ii: s_res/s_R predicts the ratio at least as well as rho: its Spearman correlation is no weaker than that of rho in absolute value, at n = 960. Refuted otherwise.
• H7l-iii (one threshold across systems): a single cut on s_res/s_R classifies win (ratio < 1) versus not at n = 960 with accuracy of at least 90% over the 42 pairs. Refuted otherwise (the systems differ in more than s_res/s_R, e.g. the twin-mean interval or the score range).
• Prior: H7l-i yes; H7l-ii roughly tied (rho and s_res/s_R are linked when sds are similar); H7l-iii partial, since cart-pole's smaller R spread makes the range term matter differently.
• Not tested: causal claims, real pairs, other N.

### Outcome (42 pairs per n; experiments/m7l_predictor/results/; analysis of the 7j and 7k CSVs, no new simulation)

• H7l-i refuted. Pooled Spearman(ratio, s_res/s_R) is 0.09 at n = 960 (0.11 at n = 240), against a threshold of 0.8 and with the wrong sign. Within the pendulum, the 21 pairs' rho runs from 0.17 to 0.88 and s_res/s_R from 0.48 to 1.09, yet the ratio sits between 0.65 and 0.89 for 20 of 21 (only the worst pair, delay-3 with rho 0.17, is above 1, at 1.27).
• H7l-ii refuted. |Spearman| is 0.09 for s_res/s_R and 0.17 for rho: neither predicts the ratio, and rho is the (slightly) better of two poor predictors.
• H7l-iii refuted. The best single cut on s_res/s_R classifies win versus not with 73.8% accuracy at n = 960 (cut 0.575; 61.9% for rho), below the 90% threshold.
• The 7k reading "cart-pole loses because its pairs have lower rho" is not supported: cart-pole rho (0.37 to 0.86) overlaps the pendulum's, and rho does not predict the win in either system.
• Post hoc, not pre-registered: the real-only interval width predicts the win far better. Spearman(ratio, real-only width) is -0.86 at n = 960, and one cut on real-only width gets 97.6% accuracy (90.5% at n = 240): the twin pays when the real-only interval is wide. Cart-pole's real-only width at n = 960 is 0.0072 (median) against the pendulum's 0.0278, so the fixed twin-mean interval (about 0.002 at N = 20,000, 7k) is about 28% of the cart-pole real-only width and 7% of the pendulum's. Caveat: ratio has the real-only width in its denominator, so part of this correlation is mechanical; the claim to test is the threshold form (win when the real-only width exceeds a multiple of the twin-mean interval width), with the twin-mean width varied directly.
• Reading: pair quality alone does not explain the finite-twin win; the absolute scale of the real-only interval against the fixed twin-mean cost does. Caveats: two systems, iid-from-pool twin draws, model-versus-model pairs, no real data, the post hoc finding is a hypothesis.
• Next: milestone 7m, pre-register the threshold form (win iff w_real > k times the twin-mean width, k fitted on pendulum, tested on cart-pole) by varying N and n on the cart-pole pairs, then move to the paper draft (milestone 7 still thin: 74 lines).

2026-10-25 · Milestone 7k: does cart-pole ever win with a large twin pool? (planned before running)

• Why: 7j showed the finite-twin win becomes typical on the pendulum (20 of 21 at n = 960) once N = 20,000; cart-pole won on 0 of 21 at N = 4,000 (7i) and only 4 of 21 even with an exact twin mean (7h). Is that a pool-size effect or a property of the pairs (lower rho)? Code: experiments/m7k_cartpole_large_n/large_n.py (7j's construction, cart-pole score pairs from m7, N in {20,000, 100,000}, f in {0.1, 0.5}, n in {240, 960}, 200 replications, bank of 20 twin intervals per (N, f); the interval is accumulated in row chunks to fit N = 100,000). Draws are iid with replacement from the 4,000-context pool, so the target is the pool mean and this is about interval width only.
• H7k-i: at N = 100,000, f = 0.1, n = 960 the combined interval is narrower than real-only on at least 3 of 21 cart-pole pairs. Refuted otherwise (the cart-pole limit is the pairs, not the pool).
• H7k-ii: the number of winning pairs at N = 100,000 is at least that at N = 20,000 in every (n, f) cell (monotone in N).
• H7k-iii: coverage at least 95% in every cell.
• Prior: 7h's exact-mean ceiling (4 of 21 at n = 960, min ratio 0.86) bounds the finite-pool result, so I expect H7k-i to fail or barely pass.

### Outcome (21 cart-pole pairs, 200 replications per cell; experiments/m7k_cartpole_large_n/results/)

• Bug caught first. My first full run (coarse 1/600 grid, as in m7h) gave combined intervals 2 to 15 times wider than real-only at N = 100,000, growing with N. Cause: at N >= 20,000 the cart-pole twin-mean interval is narrower than the grid step (width 0.0018 against a step of 0.0017), and at N = 100,000 the grid missed the interval entirely in 1 of 20 (f = 0.1) and 4 of 20 (f = 0.5) bank draws, triggering the (0.5, 1.0) fallback. The reported run uses a fine grid centred on the twin sample mean (interval_chunked; half-width 0.01, widened until the edges are rejected). Twin-bank widths are now 0.0019 and 0.0017 (N = 20,000) and 0.0010 and 0.0008 (N = 100,000). Check on 7j: its pendulum twin intervals are 0.006 to 0.013 wide, 4 to 8 grid steps, and the grid rounds outward, so 7i and 7j are at worst slightly conservative; their conclusions stand.
• H7k-i refuted (narrowly). At N = 100,000, f = 0.1, n = 960 the combined interval is narrower than real-only on 2 of 21 pairs (threshold 3); median ratio 1.222, best 0.917. Cart-pole does not become typical with a larger pool.
• H7k-ii supported. Winning pairs never fall as N rises from 20,000 to 100,000: 0 to 1 (n = 240, f = 0.1), 2 to 2 (n = 960, f = 0.1), 0 to 2 (n = 960, f = 0.5), 0 to 0 (n = 240, f = 0.5). Median ratios fall slightly (1.361 to 1.222 at n = 960, f = 0.1).
• H7k-iii supported. Minimum coverage 98.0%.
• Reading: the twin-mean interval is already negligible at N = 20,000 (about 0.002 wide), so the cart-pole limit is the residual interval of the pairs (lower rho), not the twin pool. More twin runs cannot fix it; a better twin (higher rho) can. Pendulum and cart-pole now differ by pair quality, not pool size.
• Caveats: the 20,000 or 100,000 draws are iid from a 4,000-context pool (target: the pool mean); bank of 20 twin intervals; model-versus-model pairs; no real data.
• Next: milestone 7l, relate the win to rho and to the residual variance across both pools (a fitted threshold for when a twin pays), and a fresh twin pool of distinct contexts.

2026-10-25 · Milestone 7j: does a larger twin pool or an uneven alpha split make the finite-twin win typical? (planned before running)

• Why: 7i's win (8 of 21 pendulum pairs at n = 960, N = 4,000, even 0.025/0.025 split) is limited by the twin-mean interval, whose width does not shrink with n. Twin runs are cheap, so test N = 20,000, and give the twin-mean interval a different share f of alpha (f = 0.1 or 0.5 of 0.05). Code: experiments/m7j_n_sweep/n_sweep.py; pendulum pairs only (cart-pole won on 0 of 21 at N = 4,000), n in {240, 960}, 200 replications, bank of 40 twin intervals per (N, f). N = 20,000 exceeds the 4,000-context pool so draws are iid with replacement from its empirical distribution (target the pool mean); this is a statement about the interval's width, not about 20,000 distinct twin contexts.
• H7j-i: at N = 20,000, f = 0.5, n = 960 the combined interval is narrower than real-only on at least 12 of 21 pairs. Refuted otherwise.
• H7j-ii: at N = 4,000, n = 960 the median ratio is lower with f = 0.1 than with f = 0.5 (a small twin share is better because the residual interval carries the larger term). Refuted otherwise.
• H7j-iii: coverage at least 95% in every cell.
• Not tested: cart-pole, f beyond two values, joint capital, real pairs.

### Outcome (21 pendulum pairs, 200 replications per cell; experiments/m7j_n_sweep/results/)

• H7j-i supported. With N = 20,000 twin draws and n = 960 the valid combined interval is narrower than real-only betting on 20 of 21 pairs for both f = 0.5 (median ratio 0.861, best 0.674) and f = 0.1 (0.835, best 0.655); at n = 240, 11 of 21 (f = 0.5) and 18 of 21 (f = 0.1). The 7h win largely returns once the twin pool is about 5 times the 7i one: the finite-pool cost was the twin-mean interval.
• H7j-ii refuted at n = 960, supported at n = 240. At N = 4,000, n = 960 the median ratio is 1.076 (f = 0.1) against 1.064 (f = 0.5), no gain; at n = 240 it is 1.034 against 1.087 and wins rise from 7 to 9 of 21. At N = 20,000 the smaller twin share helps at n = 240 (18 against 11 of 21) and is about equal at n = 960.
• H7j-iii supported. Minimum coverage 98.5% (conservative).
• Reading: on the pendulum pairs a valid twin-assisted certificate shorter than real-only is typical (20 of 21 at n = 960) with about 20,000 twin runs, and absent at 500. Caveats: the 20,000 draws are iid from a 4,000-context empirical pool (interval width, not 20,000 distinct contexts; target is the pool mean); pendulum only; model-versus-model pairs; no real data.
• Next: milestone 7k, cart-pole at N = 20,000 and 100,000, and a fresh twin pool of more than 4,000 distinct contexts so the target is not the pool mean.

2026-10-25 · Milestone 7i: does the twin-assisted betting win survive a finite twin pool? (planned before running)

• Why: 7h's pendulum win (20 of 21 pairs at n = 240 and 960) treated the twin mean as exact. A user has a finite number N of twin runs, so the twin mean needs its own interval, and the two intervals must be combined with a valid error budget.
• Construction (experiments/m7i_finite_twin/finite_twin.py): same pools, betting construction (c = 0.99) and n in {60, 240, 960} as 7h. Twin side: N iid draws (with replacement from the 4,000-context twin column, target the pool twin mean) for N in {500, 4,000}, a betting interval at level 0.025. Pair side: betting interval for the rescaled residual at level 0.025. Combined interval = Minkowski sum, valid at 0.05 by the union bound. Compared with real-only betting at 0.05. A bank of 200 twin intervals per N is drawn once and sampled per replication (the twin draws are independent of the real draws, so this stays valid). 300 replications per cell.
• H7i-i (survives, N = 4,000): the combined interval is narrower than real-only betting on at least 10 of 21 pendulum pairs at n = 960. Refuted otherwise.
• H7i-ii (N matters): at N = 500 the number of winning pendulum pairs at n = 960 is smaller than at N = 4,000. Refuted otherwise.
• H7i-iii (validity): coverage of the true pool mean at least 95% in every cell. Refuted otherwise (bug).
• Prior: the 0.025 split and twin interval cost a little; I expect the win to shrink (median ratio near 0.7 to 0.8 at n = 960) but survive on the high-rho pairs. Cart-pole not expected to win.
• Not tested: real pairs, other splits of alpha, joint-capital constructions.

### Outcome (2 x 21 pools, 300 replications per cell; experiments/m7i_finite_twin/results/)

• H7i-i refuted (narrowly). With N = 4,000 twin runs the combined interval is narrower than real-only betting on 8 of 21 pendulum pairs at n = 960 (threshold was 10), 7 of 21 at n = 240, 1 of 21 at n = 60. The best pair's ratio is 0.80 at both n = 240 and 960; the median pair is 1.10 and 1.05, i.e. slightly wider. The exact-twin-mean figures of 7h (20 of 21, median 0.57 at n = 960) were therefore mostly the exact-mean assumption: a finite pool costs the 0.025 split and the twin interval's own width, which does not shrink with n.
• H7i-ii supported. With N = 500 the combined interval wins on 0 of 21 at n = 960 (1 of 21 at n = 240; median ratio 1.7 at n = 960, because the twin interval at alpha/2 is wider than the whole real-only interval gain).
• H7i-iii supported. Minimum coverage 99.0% (pendulum) and 99.7% (cart-pole) over all cells; conservative.
• Cart-pole: 0 of 21 in every cell (best ratio 1.05 at n = 960, N = 4,000).
• Reading: a twin-assisted betting certificate beats real-only betting on the highest-rho pendulum pairs by up to 20% only when the twin pool is of order 4,000 runs or more and n is a few hundred or more; with 500 twin runs it never does. Twin runs are cheap, so the next question is the N at which the win becomes typical (N = 20,000 to 100,000), not the construction.
• Limitations: a union-bound split of 0.025/0.025 (not optimised; a joint capital or an uneven split could do better); twin draws are iid from the pool's empirical distribution; model-versus-model pairs; one betting construction; no real data.
• Next: milestone 7j, sweep N in {500, 4,000, 20,000, 100,000} with the twin interval on a coarser grid, and an uneven alpha split.

2026-10-25 · Milestone 7h: was the negative certificate result an artefact of the capped bet? (planned before running)

• Why: every negative certificate result (3c to 3e, 7e, 7g) rests on range-limited bounds. m3d tried a betting interval but used the paper-default cap of 1/2 over the candidate mean on the bet, and measured that the cap bound on 99 to 100% of bets; it noted "cap 0.9 with a positivity guard" as untried. Positivity of the hedged capital needs only lambda < 1/m (plus side) and lambda < 1/(1-m) (minus side), so a cap of c/m with c = 0.99 is valid. If the cap, not the range, hid the twin's variance gain, the paper's claim "the twin never shortens a certificate at these n" is wrong in a way a reviewer would find.
• Construction (experiments/m7h_uncapped_betting/uncapped.py): hedged-capital betting interval (Waudby-Smith and Ramdas 2024, JRSS-B, PrPl-EB bets, theta = 1/2, final capital at least 1/alpha rejects) on a 600-point grid, with cap constant c in {0.5, 0.99}. Real-only: values in [0, 1], alpha = 0.05. Twin-assisted, favourable case as in 7g (twin mean treated as exact): residual z = R - T rescaled from [-1, 1] to [0, 1], alpha = 0.05, plus the exact twin pool mean. Pools: the 21 pendulum pairs (m3) and 21 cart-pole score pairs (m7); n in {60, 240, 960}; 300 replications per cell (coverage is expected near 100% so more are not needed for it; widths are averages over replications); samples drawn without replacement from the 4,000-context pool.
• H7h-i (the cap was binding): with c = 0.99 the real-only betting interval is narrower than with c = 0.5 on every pair at n = 240, and narrower than real-only empirical Bernstein (m7g) on at least 18 of 21 pendulum pairs. Refuted otherwise.
• H7h-ii (a crossover exists at modest n): with c = 0.99 and the exact twin mean, the twin-assisted betting interval is narrower than real-only betting on at least 3 of 21 pendulum pairs at n = 960. Refuted if fewer than 3.
• H7h-iii (validity): coverage at least 95% in every cell. Refuted otherwise (would point to a bug).
• Prior: H7h-i yes, H7h-ii yes on the highest-rho pairs, since removing the range term leaves the variance ratio. If it holds, the programme's "never" must be narrowed, and the favourable exact-twin-mean case needs a valid version (twin mean from a finite pool) before any claim.
• Not tested: a valid combined bound with a finite twin pool, real pairs, cap constants other than the two.

### Outcome (2 x 21 pools, 300 replications per cell, n in {60, 240, 960}; experiments/m7h_uncapped_betting/results/)

• H7h-i half supported. Real-only betting with c = 0.99 is narrower than real-only empirical Bernstein on 21 of 21 pairs in both pools at n = 60 and 240 (median width ratio 0.28 and 0.45 on the pendulum, 0.26 and 0.28 on cart-pole), and on 18 and 21 of 21 at n = 960. It is not narrower than c = 0.5 on every pair at n = 240: on the pendulum, 10 of 21 (median ratio 1.006) and at n = 960 on 0 of 21 (median 1.129), because the cap stops binding as n grows and the larger cap then costs a little; on cart-pole c = 0.99 wins on 21 of 21 at n = 240. So the cap did bind where the outcome sd is small relative to the range, but the larger effect is simply EB's range term.
• H7h-ii supported, and it reverses 7g's reading. With c = 0.99 and the twin mean treated as exact, the twin-assisted betting interval is narrower than real-only betting on 20 of 21 pendulum pairs at both n = 240 (median ratio 0.78, min 0.59) and n = 960 (median 0.57, min 0.49), and on 4 of 21 at n = 60 (median 1.22). On cart-pole it needs more: 1 of 21 at n = 240 (median 1.37), 4 of 21 at n = 960 (median 1.07, min 0.86). With c = 0.5 the gain appears only at n = 960 on the pendulum (20 of 21, median 0.79). So "the twin never shortens a certificate at these n" was an artefact of the empirical-Bernstein range term and the capped bet; with a variance-adaptive uncapped bet the pendulum crossover is between n = 60 and 240.
• H7h-iii supported. Minimum coverage 98.7% (pendulum) and 99.7% (cart-pole) over all cells and variants (300 replications; conservative, as expected).
• Caveats that bound the claim. (1) The twin mean is treated as exact; the valid version adds a confidence interval for the twin mean from N = 4,000 twin runs (a betting interval on a twin sd near 0.1 is about 0.01 wide, a sizeable share of a twin-assisted width of order 0.02 at n = 960), so the exact-mean win is an upper bound on what a user gets and the valid combination is the next step. (2) Rho is 0.85 to 0.88 on the winning pendulum pairs; the cart-pole pairs, whose residual is less concentrated, barely win. (3) Coverage near 100% shows the intervals are still conservative; widths are those of a conservative method. (4) Pools of 4,000, model-versus-model pairs, one betting construction (PrPl-EB, theta = 1/2), 300 replications.
• Consequence for the programme: the paper's certificate negative must be narrowed from "never" to "not with empirical-Bernstein or capped-bet intervals"; the corrected question is whether a valid twin-assisted betting bound (finite twin pool, union bound or joint capital) keeps the win.
• Next: milestone 7i, the valid combination with a finite twin pool at N in {500, 4,000}.

2026-10-25 · Milestone 7g: does the twin shorten a certificate at larger n? (SureSim check; planned before running)

• Why: a related paper missing from our related work, SureSim (Badithela et al., arXiv:2510.04354, ICRA 2026; read from its abstract and search snippets only, the full text was unreachable through the proxy), also builds on prediction-powered inference with non-asymptotic intervals and reports a 14.4% narrower interval than real-only at N = 700 (width 0.16 against 0.187). Our negative result stops at n = 240, which the paper names as a limitation. If twin-assisted bounds win at n of several hundred, the claim "never tighter" must be narrowed to a crossover.
• Construction (experiments/m7g_large_n/largen.py): fixed-weight PPI residual z = R - T (lambda = 1, no pilot split, so no n/3 spent on fitting lambda), empirical-Bernstein two-sided interval on z with range 2 and delta/2, plus an EB interval for the twin mean over the whole 4,000-context twin pool (delta/2), against real-only EB (range 1, delta). n in {240, 480, 960, 1920}; pools: the 21 pendulum pairs (m3) and the 21 cart-pole score pairs (m7). Samples are drawn without replacement from the 4,000-context pool, so intervals are conservative for the finite population at n = 1920.
• H7g-i (crossover exists): at n = 960 the PPI-EB interval is narrower than real-only on at least 3 of 21 pendulum pairs (the highest-rho ones). Refuted if fewer than 3.
• H7g-ii (validity): coverage of the pool mean at least 95% in every cell. Refuted otherwise.
• H7g-iii (rho orders the crossover): among pairs that cross by n = 1920, the smallest winning n falls with rho (Spearman <= -0.8). Refuted otherwise.
• Same three on the cart-pole score pairs, reported as replication.
• Not tested: real pairs (still the main gap), SureSim's own algorithms (its text was not read), finite twin pools smaller than the real sample.

### Outcome (2 x 21 pools, 1,000 replications per cell; experiments/m7g_large_n/results/)

• H7g-i refuted. The fixed-weight PPI empirical-Bernstein interval (N = 4,000 twin runs) is narrower than real-only on 0 of 21 pendulum pairs at every n up to 1,920 (median width ratio 1.80 to 1.89) and on 0 of 21 cart-pole pairs (2.28 to 2.57). Even with the twin mean treated as exact the ratio is 1.25 at n = 960 and 1.04 to 1.13 at n = 1,920 on the pendulum (min 1.042), still above 1; on cart-pole it is 1.62 (min 1.29). The ratio with N = 4,000 does not fall with n because the twin-mean term does not shrink; the real-pairs term does, so the gap is the twin-side budget plus the doubled range.
• H7g-ii supported, trivially. Coverage is 100% in every cell (EB is conservative, and sampling without replacement from a finite pool adds to it).
• H7g-iii untestable: no pair crosses within n <= 1,920, so the Spearman test has nothing to rank.
• Added after the 20-replication smoke run (disclosed): the exact-twin-mean variant and a plug-in analytic crossover n* from population standard deviations (not simulated). With the twin mean exact, the smallest n at which the PPI half-width falls below real-only is 2,567 on the best pendulum pair (median 4,112, max 45,445, one pair never) and 53,096 on the best cart-pole pair (median 95,541). None is at or below 700.
• Reading: "the twin never shortens a certificate" narrows to "not at n up to about 2,000 here; a crossover exists at n of several thousand on the pendulum and tens of thousands on cart-pole, where real trials are not the scarce resource". SureSim's reported 14.4% narrower interval at N = 700 is therefore not reproduced by this construction on these pairs; the cause is not established (their rectifier, interval and tasks differ, and I have not read their text). It may be a higher paired correlation or a tighter interval method than EB with a union-bound range term.
• Limitations: one interval family (Maurer-Pontil EB) with a worst-case range term; finite pools of 4,000; model-versus-model pairs; SureSim's construction not implemented.

2026-10-25 · Milestone 7f: how wrong may the twin's tail be before an upper bound on the cost fails? (planned before running)

• Why: 7d/7e certify only a lower bound on E[R] (clipped mean). An upper bound needs a tail assumption; the checkable one is "real excess over c is at most k times the twin's": E[(R-c)+] <= k E_T[(T-c)+], giving U = (upper bound on E[min(R,c)]) + k E_T[(T-c)+]. Rather than pick k, report the smallest valid k* per pair on the 21 m7c pools (experiments/m7f_tailratio/tailratio.py, c the twin q-quantile, q = 0.99 and 0.999; the full 4,000-context pools, so this is the population value, no sampling).
• H7f-i: at q = 0.999, k* <= 3 on at least 15 of 21 pairs (a modest tail inflation suffices where the omitted effect is small).
• H7f-ii: the pairs with k* > 10 at q = 0.999 are exactly the three friction pairs that failed in 7d (friction-1, -2, -3). Refuted if the set differs.
• Not tested: the finite-sample upper bound itself, a twin tail estimated from finitely many twin runs, real pairs.

### Outcome (21 cost pools, population values; experiments/m7f_tailratio/results/)

• H7f-i refuted. At q = 0.999, k <= 3 on 0 of 21 pairs (median k 7.8; k <= 10 on 13). The twin's tail mass beyond c is tiny (E_T[(T-c)+] = 0.00015 by construction, about 4 of 4,000 contexts), so any real excess is a large multiple of it. At q = 0.99, k <= 3 on 11 of 21 (median 2.97), but still 0 of 21 at k <= 1: even the no-effect pair needs k above 1 because its real contexts are independent draws, not the twin's.
• H7f-ii refuted. At q = 0.999 eight pairs have k > 10, not three: friction-0 (35), friction-1 (1,840), friction-2 (22,175), friction-3 (72,645), sensor-3 (13), gain-1 (14), gain-2 (47), gain-3 (307). The friction-1..3 pairs are in it, with k in the thousands; the rest are moderate-effect pairs.
• Reading: a tail-ratio assumption of the "real is at most k times the twin" kind is only usable with k of order 10 to 100 for most pairs and is hopeless (k* in the thousands) exactly where the twin is badly wrong, so the user would have to choose k blind. The twin's tail quantile is a poor basis for an upper bound on an unbounded cost; an upper certificate needs real tail data or a different assumption. This closes the "upper bound on cost" item from 7d/7e as negative for the twin-tail-ratio route.
• Limitations: population values on one plant; the sampling error of k* is not part of this, so these are lower bounds on the k a real user would need.

2026-10-25 · Milestone 7e: does a twin-assisted lower bound on the clipped cost beat real-only? (planned before running)

• Why: 7d's open item. With c fixed by the twin quantile, outcomes are y = min(R, c)/c in [0, 1], so the bounded-outcome constructions apply. Real data for the programme's main gap (per-episode pairs) was probed first and is not reachable here: the SimplerEnv repository (cloned, 11 MB) holds code only, no episode logs, and the PolaRiS paper (arXiv:2512.16881) reports its real-versus-sim scores in figures, not tables, so nothing could be transcribed without guessing. Logged as a negative probe.
• Construction (experiments/m7e_clip_cv/clipcv.py): on the 21 m7c pools, q = 0.999, c as in 7d. L = EB lower bound on E[y_R - y_T] over n real pairs (residual rescaled from [-1, 1] to [0, 1]; Maurer-Pontil, delta 0.05) plus E[y_T], the exact mean of the clipped twin over the 4,000-context twin pool (treated as known, as c is). Compared with the 7d real-only bound at the same n (30, 60, 120, 240), 1,000 replications, same seeds scheme.
• H7e-i (3c to 3e repeat): at n = 240 the twin-assisted bound has a smaller mean gap than real-only on fewer than 5 of 21 pairs. Refuted if 5 or more.
• H7e-ii (validity): coverage of E[min(R, c)] at least 95% on every cell. Refuted otherwise (would point to a bug).
• H7e-iii (where it wins): every pair where it wins at n = 240 has rho >= 0.9. Refuted if a winner has rho below 0.9.
• Not tested: a twin mean estimated from finitely many twin runs, per-episode real pairs, an upper bound on cost.

### Outcome (21 cost pools, q = 0.999, c = 2.009, 1,000 replications per cell; experiments/m7e_clip_cv/results/)

• H7e-i supported. The twin-assisted bound has a smaller mean gap than real-only on 0 of 21 pairs at every n (30, 60, 120, 240). Median relative gap at n = 240: 0.333 twin-assisted against 0.271 real-only (n = 30: 2.46 against 1.51). Even the pairs with clipped rho near 0.94 lose: doubling the residual range from [0, 1] to [-1, 1] costs more than the smaller residual variance saves, the same range-term effect as 3c to 3e.
• H7e-ii supported, trivially. Coverage is 100% on every cell for both bounds (conservative).
• H7e-iii vacuous. There are no winners, so the rho >= 0.9 condition has nothing to test.
• Reading: the twin-chosen clip level makes an unbounded cost certifiable (7d), but the twin adds nothing to the certificate even with its mean treated as exactly known, the most favourable case. The certification result (H3 refuted) now holds on bounded scores, binary failures and a clipped unbounded cost.
• Limitations: the twin mean is taken from the 4,000-context twin pool as known; one EB construction, one plant; no real data.
• Negative probe (real data): SimplerEnv holds code and no episode logs; PolaRiS reports real versus sim scores in figures only. Per-episode real pairs stay the main gap.

2026-10-25 · Milestone 7d: what does a twin-chosen clip level buy a certificate on an unbounded cost? (planned before running)

• Why: 7c left "certificates for unbounded outcomes" open. No distribution-free lower bound on E[R] exists without a tail assumption, but the twin is free to run, so its quantile can fix a clip level c, and a bounded bound applies to min(R, c) (valid for E[min(R, c)] <= E[R]). Question: is that bound informative, and does the twin's quantile transfer to the real outcome? On the 21 m7c cost pools (4,000 contexts), c = the q-quantile of the twin cost over all 4,000 contexts (taken as known: the twin costs nothing), q in {0.99, 0.999}; real-only Maurer-Pontil empirical-Bernstein lower bound on min(R, c)/c (delta 0.05), pool draws without replacement, 1,000 replications, n in {30, 60, 120, 240}. Code: experiments/m7d_clip/clip.py. Nothing tuned, no pool inspected beyond the m7c summary.
• H7d-i (the twin quantile does not transfer): with q = 0.99, the share of real outcomes above c exceeds 3% on at least 5 of 21 pairs. Refuted otherwise.
• H7d-ii (clipping loses little mean): with q = 0.999, (E[R] - E[min(R, c)]) / E[R] is under 5% on at least 15 of 21 pairs.
• H7d-iii (the bound is uninformative): at n = 240, q = 0.999, the median over pairs of (E[R] - L) / E[R] exceeds 0.5, since the range c is many times the mean (the m7c maximum was 22.3 against a mean 0.35). Refuted if it is 0.5 or less.
• Also reported: coverage of L for E[min(R, c)] (expected >= 95%).
• Not tested: a twin-assisted bound (3c-3e failed on bounded outcomes; the range term only gets worse here), an upper bound on cost (needs a tail assumption), real pairs.

### Outcome (21 cost pools, 1,000 replications per cell; experiments/m7d_clip/results/)

• H7d-i supported. At q = 0.99, more than 3% of real outcomes exceed the twin's clip level on 7 of 21 pairs (median 2.3%). The failures are the large-effect pairs: friction-1 60%, friction-2 99.8%, friction-3 100% (the real cost is wholly above the twin's 99th percentile), plus friction-0, gain-1, gain-2, gain-3 (3.2% to 18%). A twin quantile transfers only when the omitted effect is small.
• H7d-ii supported. At q = 0.999 the clipping bias is under 5% of E[R] on 18 of 21 pairs (median 0.23%); the three that lose are friction-1, -2, -3 (14%, 62%, 84%), the same pairs.
• H7d-iii refuted. At n = 240, q = 0.999 the median relative gap (E[R] - L)/E[R] is 0.270, not above 0.5 (n = 30: 1.51, n = 60: 0.83, n = 120: 0.47). The clipped bound is informative from about n = 120 on the median pair: the twin-chosen range is 3 to 4 times a typical outcome, not "many times". My prior was too pessimistic. Coverage of E[min(R, c)] is 100% on every cell (conservative, as for 3e).
• Reading: a free twin quantile gives a usable certificate for an unbounded cost wherever the twin is not badly wrong about the tail, and exactly where it is wrong (friction-2, -3) the bound reports a mean that is 60% to 85% short, which the user can see from the bias, not from the interval. It certifies the clipped mean, so it is a lower bound on E[R] only; an upper bound on cost still needs a tail assumption. Not a twin-assisted bound: the twin only fixes c.
• Limitations: same plant and pools as 7c; c is computed from the full 4,000-context twin pool (treated as known); real-only bound, delta not split; no real data.
• Next: per-episode real pairs remain; a twin-assisted bound with a twin-chosen range.

2026-10-25 · Milestone 7c: does it hold with an unbounded, heavy-tailed outcome? (planned before running)

• Why: the paper's limitations name "a task with an unbounded outcome" as untested. Both earlier tasks score in [0, 1]. Here the same cart-pole, contexts, seeds and 21 pairs are re-run with a different outcome: tracking cost J = integral over the full 4 s of (100 theta^2 + (x - x_ref)^2) dt, no early stop (a fall keeps accumulating, so J has a long right tail and no upper bound). Code: episode(..., outcome="cost") in m7_cartpole/pairs.py (default behaviour unchanged), pools via m7c_cost/run_pool.py, analysis m7c_cost/analyse.py = the m3 analysis unchanged (4,000 replications, n = 30 and 60, m = 9n twin runs). Nothing was tuned: no probe run, no pool looked at before this entry.
• H7c-i (coverage worsens with a heavy tail): the twin cost has skewness above 2 on the no-effect pair, and cv or ppi_pp at n = 30 fall below 93.0% on at least 10 of 21 pairs (cart-pole score: 3; pendulum: 15 and 20). Refuted if fewer than 10, or if skewness is 2 or less.
• H7c-ii (the pairing gain survives): among pairs with rho >= 0.75, at least 80% have a paired-over-unpaired variance-ratio lower bound >= 2, and unpaired E stays in [0.9, 1.0] on every pair. Untestable if fewer than 3 pairs reach rho 0.75 (then reported as such). Prior: Pearson rho is outlier-driven under heavy tails, so rho may be both lower and a worse predictor of E.
• H7c-iii (rho still predicts E): Spearman(E, rho) > 0.9 and the over-pairs interval of |Spearman(E, rho)| - |Spearman(E, KS)| excludes 0. Refuted otherwise.
• Not tested: real-robot or per-episode real pairs (gap 1), a certificate on an unbounded outcome (the floor and 3c-3f assume [0, 1]).

### Outcome (21 cost pools, rho 0.60 to 0.94, 4,000 contexts each; experiments/m7c_cost/results/)

• H7c-i refuted on its first clause, and the substance is split by estimator. Twin cost skewness is 1.89 (no-effect pair 1.85), below the 2 I pre-registered; the cost is right-skewed with a maximum of 22.3 against a mean of 0.35 but not as skewed as I guessed. Counting cv or ppi_pp, 18 of 21 pairs fall below 93.0% at n = 30, but all 18 are ppi_pp (worst 90.18%); the fitted-lambda cv never falls below 93.0% (min 93.32%, 0 of 21 pairs) and real-only is at worst 91.57%. At n = 60: cv min 94.35%, ppi_pp 91.65%. So on an unbounded cost the shortfall belongs to the fixed lambda = 1 estimator, not to pairing, which is the opposite of the pendulum where cv also failed on 15 of 21.
• H7c-ii supported. 17 pairs have rho >= 0.75; all 17 have a variance-ratio lower bound >= 2; unpaired E is 0.948 to 0.971 on every pair; paired E is 1.44 to 5.16 (the four pairs under the bar have rho 0.60 to 0.72, E 1.44 to 1.80).
• H7c-iii refuted on both clauses. Spearman(E, rho) = 0.884 (< 0.9), Spearman(E, KS) = -0.809, and the |rho| - |KS| interval is [-0.070, 0.299], which includes 0. With a heavy right tail, rho is no better a predictor of the pairing gain than distributional distance; the pendulum and score cart-pole margins do not carry over. Seven of 21 pairs have rho within 0.01 of 0.94 (ties limit rank correlations).
• Limitations: one more outcome on the same plant, same model-versus-model pairs and swept magnitudes, no certificate tested on the unbounded outcome (the 3c-3f constructions and the floor assume [0, 1]); skewness is moderate, not extreme.
• Reading: the pairing gain (H2) now replicates on three outcome/task combinations; the interval shortfall depends on the estimator (ppi_pp) and task, and the rho-as-predictor claim (H6) is not robust to a heavier-tailed outcome. The paper's H6 sentence needs this qualifier.
• Next: per-episode real pairs (gap 1) remain.

2026-10-25 · Milestone 7b: paper draft updated with the cart-pole replication (no new experiment)

• paper/main.tex gains a "second task" paragraph (H1 shortfall is task dependent; H2 and H6 replicate; the pre-registered 0.95 rank threshold missed at 0.909) and the limitations and conclusion now say two tasks. reproduce.sh lists the m7 commands. Remaining gaps: per-episode real pairs, an unbounded outcome.

2026-10-25 · Milestone 7a: does milestone 3 replicate on a second, unstable task? (planned before running)

• Why: the paper draft's gap 2. All of m3 to m6 use one stable tracking pendulum; a reviewer will ask whether H1/H2/H6 are properties of paired inference or of that task. New task (experiments/m7_cartpole/pairs.py): MuJoCo cart-pole balanced from a random start by a linear feedback law (unstable plant, a fall ends the episode), outcome = survived fraction x (1 - RMS angle / 0.25), same five omitted effects at four swept magnitudes (21 pairs incl. the no-effect pair), 4,000 contexts per pair. Baseline push noise std 1 N (3 N gave rho about 0.2 on the no-effect pair in a 200-context probe, so it was lowered to span rho; the probe is the only tuning done and used none of the pool contexts).
• Analysis: m3_benchmark/analyse.py unchanged (4,000 replications, n = 30 and 60, m = 9n extra twin runs, same thresholds), run through m7_cartpole/analyse.py.
• H7a-i (H1 replicates): coverage of cv and ppi_pp at n = 30 falls below 93.0% on at least 10 of 21 pairs (m3: 15 and 20 of 21), because the outcome is bounded and skewed. Refuted if fewer than 10.
• H7a-ii (H2 replicates): paired E over unpaired E has lower 95% bound >= 2 on the pairs with rho >= 0.75 (at least 80% of them), and unpaired E stays in [0.9, 1.0] on every pair. Refuted if either fails.
• H7a-iii (H6 replicates): Spearman(E, rho) > 0.95 and |Spearman(E, rho)| > |Spearman(E, KS)| with the over-pairs interval of the difference excluding 0. Refuted otherwise. Prior: the interval may include 0 here, since a fall creates a big mass at low R and makes KS informative.
• Not tested: real-robot or per-episode real pairs (still gap 1), a task with a non-bounded outcome.

### Outcome (21 cart-pole pairs, rho 0.37 to 0.86, 4,000 contexts each; experiments/m7_cartpole/results/)

• H7a-i refuted: H1 does not replicate as stated. cv or ppi_pp fall below 93.0% at n = 30 on 3 of 21 pairs (friction-1, friction-3, gain-3; worst 91.98% ppi_pp), not the 10 I predicted, against 15 and 20 of 21 on the pendulum. Real-only covers at 93.5% or more everywhere. The m3 undercoverage was therefore partly a property of that task's skewed bounded outcome, not of paired inference in general; the failures that remain sit on the low-rho pairs (0.37 to 0.62) where the estimated lam is noisiest. At n = 60, cv's minimum is 92.72%.
• H7a-ii supported. All 16 pairs with rho >= 0.75 have a paired/unpaired variance-ratio lower bound >= 2 (16/16); unpaired E is 0.958 to 0.983 on every pair, as on the pendulum. The 5 pairs below the bar are the five with rho <= 0.68 (E 1.08 to 1.60).
• H7a-iii refuted on the first threshold, supported on the second. Spearman(E, rho) = 0.909 (< 0.95), Spearman(E, KS) = -0.708, with |rho| - |KS| interval [0.021, 0.494], excluding 0. So rho predicts the gain better than detectability here too, with a clearer margin than the pendulum's [0.008, 0.146], but the rank correlation is lower because ties and near-ties among high-rho pairs are noisier.
• Limitations: both tasks are mine, model-versus-model, one outcome form (bounded score); magnitudes are swept, not sourced; the baseline push level was chosen from a 200-context probe to spread rho.
• Reading: the pairing gain (H2) and rho as the predictor (H6) carry over to a second, unstable plant; the H1 coverage shortfall does not, and the paper should say the shortfall is task dependent. Gap 2 of the draft is closed for H1/H2/H6; gaps 1 (real per-episode pairs), 3 and 4 remain.
• Next: per-episode real pairs; update paper/main.tex with the second task.

2026-10-24 · Milestone 6d: is |R - T| a lower-variance target than R? (planned before running)

• Why: 6c's open question. A mean-discrepancy bound L = E[T] - E|R - T| needs an estimate of E|R - T|. It is only cheaper than estimating E[R] if the per-context variance of |R - T| (or its range) is below that of R.
• Check (experiments/m6d_target_variance/target_var.py, planned before running), on the 21 m3 pools (4,000 contexts each): Var(R), Var(R - T), Var(|R - T|), and the ratio Var(|R - T|) / Var(R); also the empirical-Bernstein half-width at n = 60 and 240 for each target using its own sample range (bounded-range scaling), to see whether the width saving survives the range term.
• H6d-i: Var(|R - T|) < Var(R) on at least 15 of 21 pairs when rho is high (>= 0.8), and on fewer than 10 of 21 across all pairs, because low-rho pairs have a discrepancy as noisy as R itself. Refuted if the split is not like this (both thresholds must hold).
• H6d-ii: on one-sided pairs |R - T| = T - R, so Var(|R - T|) = Var(R - T) and the variance ratio equals the control-variate residual ratio 1 - rho-type quantity already measured in m3; checked as an identity on the pool, tolerance 1e-9.
• Not tested: per-episode real pairs (still the main gap), a valid interval for E|R - T| plus the twin mean (that is 3e).

### Outcome (21 pools; experiments/m6d_target_variance/results/, 2 tests)

• H6d-i refuted (the second threshold failed): Var(|R - T|) < Var(R) on 20 of 21 pairs overall (all 12 with rho >= 0.8), median ratio 0.30. Only delay-3 (rho 0.17) has the discrepancy noisier than R (ratio 1.19). I expected low-rho pairs to lose more often; they mostly do not, because |R - T| is bounded below by 0 and the twin's own variation cancels in the difference.
• H6d-ii supported but narrow: only 2 of 21 pairs are strictly one-sided, and there Var(|R - T|) = Var(R - T) exactly (gap 0). 6c's "one-sided" wording should read "nearly one-sided" for the other 19.
• Empirical-Bernstein half-width (own sample range) is smaller for |R - T| than for R on 18/21 pairs at n = 60 and 20/21 at n = 240. So a bound E[R] >= E[T] - (upper confidence bound on E|R - T|) costs fewer real runs than the real-only bound, at a price: it adds the one-sided slack E|R - T| - E[T - R] (zero only when the discrepancy is one-sided), which 3e's construction already pays. This is the same trade as the control variate with lam = 1, so no new estimator results; the variance gain is the reason 3e's twin-assisted gap can approach real-only but not beat it where the discrepancy is two-sided.
• Limitations: one benchmark family (model pairs, not real robots); variance computed on the 4,000-context pool, not a finite-sample interval; EB constants are the Maurer-Pontil ones at delta = 0.05.
• Next: per-episode real pairs remain the main gap for a submittable paper.

2026-10-24 · Milestone 6c: a quantile or mean discrepancy bound instead of the maximum (planned before running)

• Why: 6b's next step. The oracle maximum |R - T| is 9 to 16 times the real-only gap, so a zero-real-run bound loses. Does an assumption on a quantile, or on the mean of |R - T|, change that?
• Ceiling lemma. If P(|R - T| > D_q) <= q with D_q in [0, 1] known, and R, T in [0, 1], then E[T - R] <= E[(T - R)+] <= (1 - q) D_q + q, so L_q = E[T] - (1 - q) D_q - q is valid with no real runs. It is tight in the class (T = 1; R = 1 - D_q with probability 1 - q, R = 0 with probability q, up to the boundary case). If instead E|R - T| <= dbar is known, L = E[T] - dbar is valid (Jensen-free: E[T - R] <= E|R - T|).
• Check (experiments/m6c_quantile/quantile.py, planned before running), on the 21 m3 pools, ORACLE inputs (the true quantile and mean over the 4,000 contexts; no procedure can know them, so this flatters the twin): for q in {0.01, 0.05, 0.10} the gap_q = mean(R) - L_q, and for the mean assumption gap_mean = mean(R) - (mean(T) - mean|R - T|). Compare with the real-only empirical-Bernstein gap of 3e at n = 30, 60, 120, 240.
• H6c-i: all bounds are valid on 21/21 pairs (deterministic consequences). Refuted if any pair has mean(R) < L.
• H6c-ii: the oracle mean-discrepancy bound beats real-only EB at n = 240 on fewer than 5 of 21 pairs, because the gap of any such bound is at least mean|R - T|, which is not small even at rho 0.87 (prior: median mean|R - T| about 0.1 to 0.2 against an EB gap of 0.06). Refuted if 5 or more win.
• H6c-iii: the quantile bounds are no better than the mean bound for q at or above 0.05, since the additive q term (0.05) already uses up most of the n = 240 budget. Refuted if any q-bound wins on more pairs than the mean bound at n = 240.
• Not tested: a data-dependent estimate of the quantile with valid adjustment (the 3e route), real-robot data.

### Outcome (21 pools; experiments/m6c_quantile/results/, 3 tests)

• H6c-i supported: all four oracle bounds valid on 21/21 pairs.
• H6c-ii refuted, for a reason that is an oracle artefact. With the oracle mean E|R - T| (0.063 to 0.73, median 0.10) the bound L = E[T] - E|R - T| has gap 0.000 to 0.002 on all 21 pairs and beats real-only EB on 21/21 at every n. This is because on this benchmark the twin omits an effect that only ever lowers the real outcome, so R <= T nearly everywhere and E|R - T| = E[T - R] = the exact bias. Knowing E|R - T| is therefore as good as knowing E[R], and no procedure has it without real data: estimating a mean of |R - T| to within the 0.06 that real-only EB achieves at n = 240 takes real pairs of the same order. The result says that a mean-discrepancy assumption is worth exactly what it costs to certify, not that it is cheap.
• Quantile bounds do not pay once n is moderate: gap median 0.235 (q = 0.01), 0.144 (0.05), 0.152 (0.10); they beat real-only EB on 15, 20, 21 of 21 pairs at n = 30 but 4, 15, 17 at n = 60 and 0 at n >= 120. The additive q term plus (1 - q) D_q is a floor near 0.12 to 0.15 whatever the twin quality. H6c-iii supported (no q-bound beats the mean bound at n = 240).
• Reading with 6b: worst-case D loses (9 to 16 times the real-only gap), a quantile bound loses beyond n of about 60, and a mean bound wins only as an oracle. The zero-real-run route is closed unless D is estimated from real data, which is milestone 3e's construction; the useful open question is a cheaper estimate of E|R - T| than E[R] itself, which needs a lower-variance target than R, i.e. |R - T| variance versus R variance (not tested here).
• Limitations: oracle inputs on one benchmark whose discrepancy is one-sided; delta = 0.05; one-sided bound.
• Next: per-episode real pairs (milestone 7 gap 1) remain the main gap for a submittable paper; a variance comparison of |R - T| and R as targets.

2026-10-24 · Milestone 6b: the floor scales with a known discrepancy bound D, and a zero-real-run ceiling (theory plus an oracle check; planned before running)

• Why: milestone 6 left open an instance-wise statement involving the discrepancy. Two things follow from the same Le Cam pair, and both can be checked on the 21 m3 pools.
• Corollary (D-floor). Restrict to laws with |R - T| <= D almost surely (D in (0, 1], R and T in [0, 1]). Then any lower bound L valid at level 1 - delta over that class, using m pairs and any number of twin-only runs, has a law in the class with a perfect twin where P(L <= 1 - D eps) >= 1/2, eps = 1 - (2 delta)^(1/m). Proof: as milestone 6 with P1: T = 1, R = 1 - D with probability eps (so mu1 = 1 - eps D and |R - T| <= D), P0: R = T = 1; data are identical with probability (1 - eps)^m, and validity under P1 gives delta >= (1 - eps)^m P0(L > 1 - eps D). Take eps = eps*.
• Ceiling. If D is known in advance and the twin law is known (unlimited twin runs), L0 = E[T] - D is valid with zero real runs, gap D. So a known D buys a gap between D eps* and D, independent of m beyond the floor, and the twin helps exactly when D is below what real data alone reach (3e's cost was estimating D from the data, which is what gamma paid for).
• Check (experiments/m6b_dfloor/dfloor.py, planned before running): for each of the 21 pools take the oracle D = max |R - T| over the 4,000 contexts (an oracle: no procedure can know it; disclosed, and it flatters the twin), L0 = mean(T) - D, gap0 = mean(R) - L0, and compare with the measured real-only empirical-Bernstein gap of 3e at n = 30, 60, 120, 240 (m3e_bounded_disc/results/bounded.csv).
• H6b-i: L0 is valid on every pair (gap0 >= 0) by construction; refuted if any pair has mean(R) < L0.
• H6b-ii: even with the oracle D, L0 has a smaller gap than real-only EB at n = 240 on fewer than 5 of 21 pairs, because the max of |R - T| over 4,000 contexts is far above the typical discrepancy. Refuted if 5 or more pairs win.
• H6b-iii: D below which a zero-real-run bound beats real-only EB is D* = EB gap(n), about 0.06 at n = 240, 0.36 at n = 30; report the pairs' D against it.
• Not tested: a data-dependent D with a valid adjustment (that is 3e), real-robot D.

### Outcome (21 pools; experiments/m6b_dfloor/results/, 3 tests)

• H6b-i supported: L0 valid on 21/21 pairs (it is a deterministic consequence of |R - T| <= D).
• H6b-ii supported, more strongly than predicted: the oracle D = max |R - T| is 0.55 to 0.99 (median 0.72) even at rho 0.87, so L0 has gap 0.26 to 0.73 (median 0.60) and beats real-only empirical Bernstein on 1/21 pairs at n = 30, and on 0/21 at n = 60, 120 and 240 (mean real-only gap 0.363, 0.193, 0.106, 0.060).
• H6b-iii: the break-even D is the real-only gap, 0.054 to 0.081 at n = 240; the pools' worst-case D is 9 to 16 times that. A worst-case discrepancy bound is the wrong assumption for a twin whose typical error is small but whose tail is not: a certificate that pays needs a bound on a quantile or a mean of |R - T|, not the maximum, which then needs real data to check (the 3e route).
• Reading. Together with milestone 6: the floor is D x 2.3/m, a known D buys at most a factor D over the worst case, and a zero-run ceiling D only wins for D below the real-only gap. Both the floor and the ceiling are far from the 3e construction (gap 0.19 at n = 240), so the open question is a tail-aware discrepancy assumption, not a better concentration inequality.
• Limitations: oracle D flatters the twin and still loses; one benchmark; one-sided; delta = 0.05; the floor is a worst-case statement over one pair of laws.
• Next: replace the maximum with a quantile or an Orlicz-type tail assumption (R - T sub-Gaussian with a known proxy) and derive the matching floor; per-episode real pairs (milestone 7 gap 1).

2026-10-24 · Milestone 7: paper draft (paper/)

• What: paper/main.tex (CoRL-style article, not yet in the venue's style file), refs.bib (13 references, all pass cite_check.py) and reproduce.sh (one command per figure and table). Every number in the draft is copied from a results/summary_*.txt or the outcome entries below; no new experiment was run, so there is no new hypothesis. LaTeX was not compiled here (no TeX in the image).
• Framing chosen: the result is mixed to negative: pairing gives up to 1/(1 - rho^2) for point estimates, but no certificate built here is tighter than real-only, and the floor (milestone 6) shows part of that is forced. Written as such rather than as a positive method paper.
• Venue decision (the proposal deferred it to milestone 6): stay with CoRL as a negative-and-measurement paper; the theory is one worst-case theorem, too thin alone for AISTATS. The paper is not ready to submit. Gaps: (1) per-episode real pairs (SIMPLER logs) to test H1/H2 on real data; (2) a second task beyond the pendulum; (3) an instance-wise bound involving rho; (4) compile in the venue's style and check the page limit.
• Next: obtain per-episode real/sim pairs; add a second MuJoCo task; instance-wise lower bound.

2026-10-24 · Milestone 6: a distribution-free floor on any twin-assisted lower bound (theory; the numerical check was planned before running)

• Why: 3e, 3f and 5b all found certified procedures that the twin does not help. Is that a flaw of those constructions or a limit on any procedure? A floor tells us how much room is left.
• Theorem (floor). Let L be any lower confidence bound for mu = E[R] with P(L <= mu) >= 1 - delta for every joint law of (R, T) on [0, 1]^2, using m paired runs, any number of twin-only runs and any randomisation. Then for eps = 1 - (2 delta)^(1/m) there is a law with a perfect twin (R = T = 1) under which P(L <= 1 - eps) >= 1/2.
• Proof. P1: T = 1 and R = 0 with probability eps independently (mu1 = 1 - eps), P0: R = T = 1. The twin-only runs have the same law under both. The paired data are identical under P1 and P0 with probability (1-eps)^m (every real outcome is 1). Validity under P1 gives delta >= P1(L > 1 - eps) >= (1-eps)^m P0(L > 1 - eps). At eps = eps, (1-eps)^m = 2 delta, so P0(L > 1 - eps) <= 1/2. QED. For small eps, eps is about ln(1/(2 delta))/m.
• Tightness: on the all-ones sample the exact valid bound is delta^(1/m) (gap 1 - delta^(1/m)), so the floor is tight to within about 1.3 times at delta = 0.05 for m between 10 and 240.
• Check (experiments/m6_floor/floor.py, 3 tests): Monte Carlo of P1(data = P0 data) matches 2 delta = 0.10 at eps* for m = 10 to 1000 (0.0987 to 0.1016 with 200,000 draws).
• Reading. (i) The floor is order 1/m and holds even for a perfect twin: no procedure can certify a gap below about 2.3/m at 95% in the worst case, so a twin cannot buy the sqrt(var/m) rate down to zero and cannot reduce the 1/m term. The floor is 0.0095 at m = 240; 3e's bounded-discrepancy bound had gap 0.193 (20 times the floor) and real-only empirical Bernstein 0.060 (6 times). So 3e's failure was not forced by the problem: there is room for a better construction, though not for one that wins by more than the sqrt term. (ii) In the zero-variance corner, empirical Bernstein's lower-order term is 7 ln(1/delta)/(3(m-1)), 3.1 times the exact bound, so real-only is already within a small factor of optimal there: the twin can win only where the sqrt term dominates (variance of R high, rho high, m large). (iii) What this does not show: an instance-wise (not worst-case) lower bound involving rho. That is the open half of H3.
• Limitations: a minimax statement over one pair of laws; not a bound that depends on rho or on a checked discrepancy assumption; one-sided; delta = 0.05.
• Next: instance-wise lower bound under a bounded-discrepancy assumption (Le Cam with D-tail mass), then the paper (milestone 7).

2026-10-24 · Milestone 5b: anytime confidence sequences with the twin as control variate (H5) (planned before running)

• Why: 3d found a fixed-n betting interval with a pilot-estimated control variate about 3 times wider than real-only, because rescaling Z = (R - lam T + lam)/(1 + lam) into [0, 1] costs range. H5 is about sequential use: stop collecting real trials as soon as a decision is clear. Here the twin is free on all N = 4,000 contexts, so E[T] is known, and the betting target can be the paired difference D = R - T (mean E[R] - E[T]), whose variance is small when rho is high, which a variance-adaptive bet can exploit without a pilot (lam = 1).
• Design (experiments/m5b_anytime/anytime.py), on the 21 m3 pools, R and T known per context, contexts drawn with replacement (iid), truth = pool mean of R, nmax = 300, delta = 0.05. Three hedged-capital confidence sequences (Waudby-Smith and Ramdas 2024, predictable plug-in bets, capital for each candidate mean on a 300-point grid, a candidate is out from the first time its capital reaches 1/delta; running intersection): real (R in [0, 1]), cv1 (D = R - T mapped from [-1, 1] to [0, 1], target shifted by the known mean of T), cvlam (first 20 trials fit lam by OLS, then the sequence runs on Z with that lam fixed; the 20 count towards n). Decision task: stop at the first t at which the sequence excludes the bar b = truth - g (so the right answer is "mean R is above b"), g in {0.05, 0.10, 0.20}; reported: median and mean stopping time censored at nmax and the share that never stop.
• H5-i (anytime validity): in 1,000 replications, the share of runs in which truth is outside the sequence at any t <= 300 is at most 5.0% (+ MC s.e.) for every method and pair; the largest tolerated is 6.0%. Refuted if any cell exceeds 6.0%.
• H5-ii (value): cv1 stops sooner than real (median stopping time, g = 0.10) on every pair with rho at least 0.8 (7 pairs in the 3c table) and is no more than 20% slower on pairs with rho below 0.5. Refuted if either half fails. Prior: after 3d I expect the first half to hold only on the best two or three pairs; a bounded-range penalty (D spans [-1, 1]) is the risk.
• H5-iii (pilot versus none): cvlam is not faster than cv1 in the median over pairs (the 20-trial pilot is a cost). Refuted if cvlam is faster in the median over pairs.
• Not tested: mixture (rather than plug-in) bets, a data-dependent nmax, real-robot logs, tighter delta splits.

### Outcome (21 pairs, 1,000 replications per cell, nmax = 300, delta = 0.05; experiments/m5b_anytime/results/)

• H5-i supported. Truth left the sequence at some t <= 300 in at most 2.5% of runs for real, 0.8% for cv1, 0.6% for cvlam (limit 6.0%). Optional stopping is safe; the sequences are conservative (median miss 0.5%, 0.0%, 0.0%).
• H5-ii refuted. cv1 is slower than real on all 12 pairs with rho at least 0.8 (g = 0.10 median stopping time 72 to 76 trials against 50 to 52). Over all pairs the median stopping time at g = 0.05 / 0.10 / 0.20 is 108 / 50 / 22 for real and 144 / 74 / 39 for cv1, so the twin costs about 1.4 to 1.8 times the real trials. Correction to my pre-registration: 12 pairs, not 7, have rho at least 0.8 (the "7" came from the 3c table's n = 60 subset). The low-rho half cannot be read as written: on delay-3 (true mean near 0) the bar truth - 0.10 is below zero, so real stops at trial 1 and the ratio (79) is an artefact; on delay-2 (rho 0.37) the ratio is 1.68, over the 20% allowed.
• H5-iii supported. cvlam is slower than cv1 (median 133 against 74 at g = 0.10): the 20-trial pilot and the (1 + lam) rescaling both cost.
• Why: D = R - T spans [-1, 1], so the bounded-range penalty seen in 3c and 3d returns; at rho 0.87 the sd of D is still about half that of R, not small enough to pay for twice the range with a plug-in bet. Reading across 3c to 3f and 5b: no certified or anytime-valid procedure tested here makes the twin pay; its value stays in point-estimate variance (milestones 2 and 3).
• Limitations: model-versus-model pairs on one task; one betting construction (plug-in, no mixture), one grid, delta not tuned; iid draws with replacement from a finite pool; bars defined relative to the pool mean.
• Next: a mixture (rather than plug-in) bet that exploits the small variance of D; a range-free target (variance-adaptive bound on D from the observed range); then milestone 6 theory and 7 paper.

2026-10-24 · Milestone 5a: active selection of real trials (H4) (planned before running)

• Why: milestones 3c to 3f showed the twin cannot shorten a certificate. H4 asks about the other use: where to spend the scarce real trials. When the twin's error D = R - T depends on the context (initial state, payload, reference), sampling contexts in proportion to the predicted size of D should cut the variance of the estimate of E[R] at equal real budget.
• Design (experiments/m5a_active/active.py), on the 21 m3 pools (4,000 contexts each, regenerated from the benchmark's seed; R and T known for every context, so the target is the pool mean of R). The twin is free on all N = 4,000 contexts. Estimator for every method: theta = mean(T over all N) + (1/n) sum_i D_i / (N pi_i), D_i = R_i - T_i, contexts drawn with replacement with probabilities pi (unbiased for any pi). Methods: uniform (pi = 1/N, i.e. PPI with lam = 1), uniform_cv (OLS lam on the sample, the m2 estimator), active (stage 1: n/3 uniform trials fit gradient boosting of |D| on the six context features, cross-fitted nothing needed because stage 2 is a new draw; stage 2: n - n/3 trials from pi proportional to 0.2 mean(pred) + pred, pred clipped below at a small floor, so pi is at least 0.2/N times the mean-ratio; the two stages are combined by sample-size weights), oracle (pi proportional to |D| + 0.05 mean|D|, the unattainable ceiling that uses the real outcomes of the whole pool). Same real budget n for all, n in {30, 60, 120}, 1,000 replications.
• Heteroscedasticity measure, per pair, before the comparison: 5-fold cross-fitted R^2 of gradient boosting for |D| on the context (het_r2).
• H4-i (validity): the 95% normal interval from the weighted terms covers at least 93% for active on every pair at n = 120 (the weights are bounded; small-n coverage is not tested because 20 stage-2 trials are too few for a CLT). Refuted otherwise.
• H4-ii (value): at n = 120, active has a lower MSE than uniform on every pair with het_r2 above 0.3, and is not more than 10% worse on pairs with het_r2 below 0.05. Refuted if either half fails. Prior: gains of 10 to 30% on the few pairs where the omitted effect interacts with the initial state (delay, gain, sensor); about zero or slightly negative elsewhere, because the n/3 pilot costs more than it returns when |D| is unpredictable.
• H4-iii (cost of learning the sampler): active MSE exceeds oracle MSE by at least 25% at n = 60 on pairs with het_r2 above 0.3 (the learning cost is visible). Refuted if within 25%.
• Amendments before the full run (after a 20-replication smoke test, no result read for the hypotheses). (a) The oracle as first written (pi proportional to |D|) is degenerate: D has one sign on most pairs, so the weighted terms are constant and its MSE is about zero, which is no ceiling. It is replaced by the same gradient-boosting predictor fit on all 4,000 real outcomes (5-fold cross-fitted), i.e. the sampler with an unlimited pilot. (b) Smallest het_r2 is 0.07, so the "below 0.05" tier of H4-ii is empty; the second half becomes: on the pairs with het_r2 below 0.15, active is not more than 10% worse than uniform at n = 120 in the median over those pairs.
• Not tested: more than two stages, sequential/bandit allocation, a different predictor of |D|, real-robot pools (the SIMPLER tables have no contexts), and the interaction with lam (the active estimator fixes lam = 1).

### Outcome (21 pairs, 1,000 replications per cell, n in {30, 60, 120}; experiments/m5a_active/results/)

• H4-i refuted, narrowly. The 95% normal interval from the weighted terms covers 92.4% at worst (friction-2 and mass-2; next 92.6%, 92.7%) at n = 120, below the 93% I set. The undercoverage is small and typical of a plug-in normal interval with heavy-tailed weights; I did not test a bootstrap or t interval.
• H4-ii supported. At n = 120 active beats uniform on all three pairs with het_r2 above 0.3: MSE ratio 0.65 (delay-2, het_r2 0.62), 0.75 (delay-3, 0.56), 0.83 (mass-3, 0.47). On the 11 pairs with het_r2 below 0.15 the median ratio is 1.08 (within the 10% allowed) but the worst is 1.32. Across all 21 pairs active wins on 10 at n = 120, median ratio 1.01.
• Small n: no gain. The n/3 pilot costs more than it returns: the median ratio is 1.18 at n = 30 and 1.13 at n = 60, and active wins on 2 and 3 of 21 pairs. The gain appears around n = 120. uniform_cv (OLS lambda) is 1 to 3% worse than lambda = 1 in MSE (median), so the comparison is not distorted by the choice of lambda.
• H4-iii supported. At n = 60 on the three heteroscedastic pairs active has 1.6 to 2.6 times the MSE of the cross-fitted all-data sampler (delay-2 1.63, delay-3 1.74, mass-3 2.57): learning the sampler from 20 trials loses most of what it could win. That ceiling is itself only 0.76 to 0.82 of uniform in the median over pairs.
• Reading: H4 holds only where the twin error is strongly predictable from the initial state (het_r2 above about 0.45 here, 3 of 21 pairs) and only with at least about 120 real trials; elsewhere the safe choice is uniform sampling. Even the ceiling is a gain of about 20% in MSE (about 10% in standard error), so active selection is a refinement, not a way to make the twin pay. This agrees with 3c to 3f: the twin's value is variance reduction through pairing, and its error structure is a second-order lever.
• Limitations: model-versus-model pairs on one task; the pool is a finite population sampled with replacement; one predictor (depth-2 gradient boosting, 40 trees), one 1/3 pilot split and one mixing constant 0.2, not tuned; coverage uses a plug-in normal interval; lambda fixed at 1 for active.
• Next: milestone 5b, anytime confidence sequences (H5); a bootstrap interval for the weighted estimator; a sequential allocation instead of a fixed pilot.

2026-10-24 · Milestone 3f: failure-rate certificates, twin-informed versus Clopper-Pearson (planned before running)

• Why: 3e refuted H3 for a max-based bound on a continuous outcome, and noted H3 was originally stated for failure rates, where the discrepancy is binary and its check is exact. Failure = outcome below tau = 0.65 (the 10% quantile of the twin pool), on the 21 m3 pools. For binary outcomes the identity p = P(R fails) <= P(T fails) + P(R fails and T passes) needs no assumption about the twin, so no separate pilot is spent: the n real trials are the pairs, and the mismatch rate q10 = P(R fails, T passes) is estimated from them with an exact interval.
• Construction (experiments/m3f_failure_rate/failrate.py): total delta 0.05 split 0.025 / 0.025. Twin-informed upper bound U = CP_upper(twin failures over N = 10n runs, 0.025) + CP_upper(count of pairs with R fail and T pass among n, 0.025). Baseline: Clopper-Pearson upper bound on the n real outcomes (0.05). Both are valid by construction (union bound). Pool draws without replacement, 1,000 replications, n in {30, 60, 120, 240}. Certification of "p < eps" with eps = 0.20 happens when U < eps.
• H3f-i (validity): coverage (U >= true p) at least 95.0% on every pair and n. Refuted if any cell is below.
• H3f-ii (value, tight-twin pairs): at n = 240 the twin-informed U is smaller (mean U - p) than Clopper-Pearson on every pair with rho >= 0.98 and on none with rho < 0.90. Refuted if it fails on any rho >= 0.98 pair or wins on any rho < 0.90 pair. Prior: wins only where q10 is under about a third of p; I expect it to hold for rho >= 0.99 and fail near 0.98.
• H3f-iii (certification power): at n = 60, on pairs whose true p is at most 0.12, the twin-informed bound certifies p < 0.20 in more replications than Clopper-Pearson on at least the pairs with rho >= 0.98. Refuted otherwise.
• Not tested: tau other than 0.65, a tighter split of delta, a tolerance test that accepts a small q10 assumption instead of bounding it.

### Outcome (21 pairs, 1,000 replications per cell, tau = 0.65; experiments/m3f_failure_rate/results/)

• Correction to my pre-registration. H3f-ii was written with a "rho >= 0.98" tier from a misreading of the benchmark's correlations. The real range of rho against the twin outcome is 0.17 to 0.878 (best pair friction-0), so that tier is empty and the clause is vacuous. The substantive test, whether the twin-informed bound is ever tighter, stands and is below.
• H3f-i supported. The twin-informed bound covers at least 98.6% on every cell (min 99.2 / 99.1 / 98.9 / 98.6 at n = 30 / 60 / 120 / 240). Side finding: Clopper-Pearson's own coverage is 93.9% at n = 60 on one pair, below nominal. It is exact for a binomial but the pools are finite and drawn without replacement, and the truth is the pool's rate, so this is a pool artefact (cells at p = 1 or near it are the likely source); not investigated further.
• H3f-ii refuted (no pair is tighter). The twin-informed bound is tighter than Clopper-Pearson on 0 of 21 pairs at every n, including n = 240 on the best-correlated pairs. Mean excess over truth at n = 240: CP 0.041, informed 0.056. On friction-0 (rho 0.878, p 0.102) the gap is 0.038 against 0.049.
• H3f-iii refuted. At n = 60 the twin-informed bound certifies "p < 0.20" less often on every pair with p <= 0.12: friction-0 36.7% against 59.0%, none-0 37.4% against 60.9%, sensor-0 34.6% against 56.2%.
• Why (measured). The mismatch rate q10 = P(R fails, T passes) is 0.056 on none-0 against p = 0.101: even with no omitted effect, the "real" system's disturbance makes the real run fail where the twin passes in about 55% of the real failures (q10/p from 0.55 on the best pairs to 0.8 and up on the worst). The informed bound is about p_T + q10 plus two exact-interval slacks, so it can win only when q10 is a small fraction of p.
• How small (plug-in, breakeven.py, expected counts, twin failure rate = p, N = 10n twin runs). The informed bound beats Clopper-Pearson only if q10/p is roughly 0.04 to 0.10 (n = 240: 0.10 at p = 0.02, 0.04 at p = 0.05, 0.06 at p = 0.10; at n = 30 never, at n = 60 only at p = 0.10 with 0.08). The table rounds expected counts to integers, so single cells are coarse; the order of magnitude is the result. Read: a twin must reproduce about 90 to 95% of the real failures, not the 45% it does here, before it shortens a failure-rate certificate. This matches the worst-case result of twin-certification and now has a number.
• Reading across 3c to 3f: no tested bound that certifies (not just estimates) a quantity from twin-plus-real data beats the real-only baseline at n up to 240, for continuous or binary outcomes. What the twin buys is point-estimate variance (milestones 2 and 3). H3 as stated is refuted in this benchmark; what remains open is an explicit tolerance test (assume q10/p <= a stated value, verified on a pilot, and price the assumption), and real failure data from SIMPLER-style episode logs.
• Limitations: one task and simulator family, model-versus-model pairs; a single threshold tau = 0.65 (twin 10% quantile); equal split of delta; twin-informed bound uses the union of two exact intervals, not a joint test of the pair counts (a multinomial or trinomial test could be a little tighter).
• Next: a tolerance test; milestone 5 (active selection H4, anytime H5), where the twin's value is variance and not certification.

2026-10-24 · Milestone 4: SIMPLER's paired real and sim success rates (planned before running)

• Why: milestones 3 to 3e used model-versus-model pairs. The first real test is SIMPLER (Li et al. 2024, arXiv:2405.05941), whose Tables IV and V give, for the same policies and tasks, real-robot success rates and SAPIEN "Visual Matching" simulated success rates. Question: for a new policy, how well does a calibrated twin predict real success, in real-trial units: how many real trials is the calibrated twin worth, and does it beat the real trials the paper actually ran?
• Data (experiments/m4_simpler/data.py, transcribed from the PDF, checked by assertions): Visual Matching only. Google Robot: 6 policies x 4 task columns (Pick Coke Can average over three orientations, n_real = 75; Move Near average, n = 60; Open Drawer, n = 27; Close Drawer, n = 27; counts from Appendix B). WidowX + Bridge: 3 policies x 4 tasks (success columns), n = 24 each. Check: every real rate times n_real is an integer within rounding (a transcription test). Sim rates are averages over many more episodes and are treated as noise-free; this is stated as a limit. Only policy-level rates are published, not episodes, so this milestone cannot test the paired control variate (H1, H2) itself; it tests the quantity that decides whether the control variate pays, the policy-level discrepancy.
• Predictors for the real rate of a held-out cell (task, policy), leave-one-cell-out: raw (sim rate), cal (OLS real ~ sim fitted on the other cells of the same robot, pooled over tasks, clipped to [0, 1]). Debiased MSE of a predictor = mean (pred - R_hat)^2 minus mean of the unbiased binomial variance of R_hat, R_hat(1 - R_hat)/(n - 1) (the held-out real rate is itself noisy). Real-only variance at the paper's n = mean R_hat(1 - R_hat)/(n - 1). Break-even n* = mean p(1-p) / MSE_cal. Effective real trials gained by an inverse-variance combination: n x v_real / m_cal (m = debiased MSE of cal). 95% intervals: bootstrap over cells (2,000 resamples; cells within a policy and task are dependent, so the intervals are optimistic).
• H4-i (the raw twin is not a substitute): debiased MSE of raw exceeds the real-only variance at the paper's n on the pooled cells. Refuted if it is below. Prior: supported by a wide margin (several cells have sim - real gaps of 0.2 or more).
• H4-ii (the calibrated twin is worth a handful of trials): cal debiased MSE is above the real-only variance at the paper's n (so the paper's real trials are worth more than a calibrated twin alone), and n is between 5 and 40. Refuted if cal beats real-only at the paper's n, or n falls outside 5 to 40. Prior: the calibration error is dominated by task-specific bias (drawer tasks), so n* is near 10.
• H4-iii (rank versus level): the pooled Pearson r between sim and real is above 0.8 while the debiased RMSE of cal stays above 0.1. SIMPLER's own metrics (MMRV, Pearson r) are rank-based, and rank agreement is the thing a level estimate does not need. Refuted if either half fails.
• Not tested: Variant Aggregation sims, Isaac Sim, per-episode pairs, the control variate with an estimated lam on real episodes (needs the released SIMPLER episode logs; not obtained).

### Outcome (36 cells: 8 task columns, 6 Google Robot and 3 WidowX policies; experiments/m4_simpler/results/)

• Data note. The real eggplant rates (.233, .433, grasp .033) are multiples of 1/30, not 1/24 as Appendix B says, so n_real = 30 for that column (a transcription test enforces integer counts; the paper's text and table disagree).
• H4-i supported. Debiased MSE of the raw sim rate is 0.0212 [0.0134, 0.0297], five times the real-only variance at the paper's n (0.0042). 10 of 36 cells have |sim - real| >= 0.2 (max 0.31, Move Near Octo-Base).
• H4-ii supported. A leave-one-cell-out calibrated twin has debiased MSE 0.0138 [0.0054, 0.0248] (RMSE 0.118), above the real-only variance at the paper's n, so the paper's real trials beat the calibrated twin alone. Break-even n* = 10.1 real trials [5.1, 26.5]: with fewer real trials than that a calibrated twin predicts a new policy's success better, with more it does not. Inverse-variance pooling would add about 0.30 [0.15, 0.79] of n in effective real trials (plug-in, assumes the calibration error is independent of the real noise). By task, calibration helps on pick-coke-can, open-drawer and eggplant (MSE below the real-only variance or close) and hurts on carrot, close-drawer and spoon.
• H4-iii supported, narrowly. Pooled Pearson r between sim and real is 0.900 [0.837, 0.948] while the calibrated RMSE is 0.118; the bootstrap interval for the RMSE [0.074, 0.158] reaches below 0.1. Rank agreement is excellent (SIMPLER's own claim) while the level error is the size of a real-trial standard error at n of about 15.
• Reading: on real data the policy-level discrepancy is about 0.12 even after calibration, so the 1/(1 - rho^2) control-variate gain does not transfer: the gain depends on the per-episode paired correlation, which the published tables cannot give. What they do show is that a SIMPLER-like twin is worth roughly 10 real trials, not 100. Next: the per-episode pairs from the released SIMPLER code (needs GPU rendering or logs; not available here) would test H1 and H2 directly.
• Limitations: policy-level rates only (no episodes, so no paired control variate); sim rates treated as noise-free; transcription from a PDF by text extraction, spot-checked with an integer-count test and three paper values; cells clustered, so the intervals are optimistic; one simulator (SAPIEN), Visual Matching only; 36 cells from 9 policies and 8 tasks.

2026-10-24 · Milestone 3e: a bounded-discrepancy lower bound (planned before running)

• Why: 3c and 3d showed range-based bounds on R - lam T cannot see the twin's variance gain. Route (b) puts the twin to work through a checked bound on the discrepancy D = R - T: if |D| <= b with b far below 1, Hoeffding on D has range 2b, not 1 + lam. The bound b must be checked on a pilot, and the check has a price. Question: at what total number of real trials (pilot included) does a one-sided lower confidence bound on E[R] built this way beat the real-only empirical-Bernstein bound?
• Construction (experiments/m3e_bounded_disc/bounded.py), outcomes in [0, 1], 21 m3 pools, total miscoverage delta = 0.05 split 0.01 / 0.02 / 0.02. Pilot = first k = n/3 real+twin pairs: b = max |D| over the pilot; then P(|D| > b) <= gamma = 1 - 0.01^(1/k) with confidence 0.99 (distribution-free, order statistics), and clipping D at b biases its mean by at most E[(|D| - b)+] <= gamma (|D| <= 1). Main pairs (n - k): Hoeffding one-sided (delta 0.02) on clip(D, -b, b), range 2b. Twin mean: Hoeffding one-sided (delta 0.02, range 1) on all N = 10n twin runs (the n pairs inside them). L = mean(T_N) - h_T + mean(clip D) - h_D - gamma. Baseline: Maurer-Pontil empirical-Bernstein one-sided bound on the n real outcomes (delta 0.05). Pool draws without replacement, 1,000 replications, n in {30, 60, 120, 240}.
• H3e-i (validity): coverage of both lower bounds is at least 95.0% on every pair and n. Refuted if any cell is below (a bug or wrong range).
• H3e-ii (value, the proposal's H3): the bounded-discrepancy bound is tighter (mean truth - L smaller) than real-only EB at n = 240 on every pair with rho >= 0.85, and not tighter at n = 30 on any pair (gamma at k = 10 is 0.37). Refuted if it is tighter at n = 30 anywhere or fails to be tighter at n = 240 on any rho >= 0.85 pair. Prior: the crossover sits between n = 60 and n = 240 and only on the highest-rho pairs; gamma, not the twin, is the binding term.
• Not tested: variance-adaptive bounds on D, a quantile (not max) bound for b, tuned delta split.

### Outcome (21 pairs, 1,000 replications per cell, n in {30, 60, 120, 240}; experiments/m3e_bounded_disc/results/)

• H3e-i supported, trivially. Both lower bounds cover 100.0% on every one of the 84 cells. Valid, and conservative.
• H3e-ii refuted, on its second half, and so is the proposal's H3 in this benchmark. The bounded-discrepancy bound is tighter than real-only empirical Bernstein on 0 of 21 pairs at every n, including n = 240 and the seven pairs with rho at least 0.85. Mean gap (truth minus bound): real-only 0.363 / 0.193 / 0.106 / 0.060 at n = 30 / 60 / 120 / 240; bounded 0.644 / 0.425 / 0.282 / 0.193. The first half (not tighter at n = 30) holds. My prior of a crossover between n = 60 and 240 was wrong.
• Why (measured). The pilot maximum of |D| stays large: mean b is 0.26 to 0.36 at n = 240 even for rho of 0.85 to 0.88 (up to 0.97 on the delay-3 pair), because the Ornstein-Uhlenbeck disturbance and the omitted effects make the discrepancy heavy-tailed even when the correlation is high. A range of 2b near 0.5 to 0.7 is no better than the real outcome's range, so Hoeffding on D gains nothing, while the check on b costs gamma = 1 - 0.01^(1/k) (0.369 at n = 30, 0.056 at n = 240) and the twin-mean term costs h_T. At n = 240 the gamma term alone (0.056) already equals real-only EB's whole gap.
• Reading across 3c, 3d, 3e: for outcomes in [0, 1] with sd near 0.1 and a max-based discrepancy bound, no certified bound built from the worst case beats real-only Bernstein at n up to 240. The twin's variance gain (milestones 2 and 3, within 6% of 1/(1 - rho^2)) is real for point estimates and cannot be certified distribution-free at these n by these constructions. H3 as stated in the proposal is refuted for a max-based b. Open: a quantile bound for b with a bias term that uses E[(|D| - b)+] estimated rather than bounded by gamma (needs a second assumption on the tail of D), variance-adaptive bounds on D, and failure-rate (binary) outcomes, where Clopper-Pearson is the baseline and the proposal's H3 was originally stated.
• Limitations: one outcome and task, model-versus-model pairs, max-based b and a fixed 0.01/0.02/0.02 split, Hoeffding rather than Bernstein on D and T.
• Next: milestone 4 (SIMPLER paired tables, real data), and a binary-outcome version of this test against Clopper-Pearson.

2026-10-24 · Milestone 3d: a betting interval (planned before running)

• Why: 3c showed empirical-Bernstein intervals hide the twin's variance gain behind their range term. A betting (hedged-capital) confidence interval (Waudby-Smith and Ramdas 2024, JRSS-B) adapts to the observed variance, so it may expose the gain while staying finite-sample valid. Question: does a betting interval for the control-variate estimand beat a betting interval on real data alone, on the same 21 pools?
• Construction (experiments/m3d_betting/betting.py): scale values to [0, 1]; for each candidate mean m on a 400-point grid, run a hedged capital process with predictable plug-in bets (lambda_t = min(sqrt(2 log(2/delta) / (v_{t-1} t log(t+1))), 0.5/m style cap, per the paper's PrPl-EB recipe), reject m when the mixed capital reaches 1/delta (the first draft of this entry said 2/delta, a slip: delta is already the total miscoverage; corrected before any full run, see outcome); interval = unrejected set. Real-only: betting CI on all n real outcomes, delta = 0.05. Control variate: first n/3 pairs give lam (OLS slope clipped to [0, 1]); betting CI on R - lam T over the other pairs (delta = 0.025, rescaled from [-lam, 1]); betting CI on T over all N = 10n twin runs (delta = 0.025); sum of the two, union bound. Observation order is randomly permuted per replication. Same pools, n in {30, 60}, 1,000 replications.
• H3d-i (validity): coverage at least 95.0% for both intervals on every pair and n (MC s.e. about 0.5 points; betting intervals are usually slightly conservative). Refuted if any cell is below 95.0%, which would point to a bug.
• H3d-ii (value): control-variate width over real-only width is below 1 on every pair with rho >= 0.85 at n = 60, and above 1 on pairs with rho < 0.3. Refuted if either half fails. Prior: the variance gain should now show, but the union-bound split and the pilot third cost width, so I expect the ratio near 1 at n = 30 and below 1 only at the highest rho.
• Not tested: tuned delta split, a larger pilot share, a joint (non-union) bound.

### Outcome (21 pairs, 1,000 replications per cell, n in {30, 60}; experiments/m3d_betting/results/)

• H3d-i supported, trivially. Coverage is 100.0% for the control-variate interval on every cell and at least 99.5% for real-only (minimum over 42 cells), at both variance priors tried (0.25, the paper default, and 0.005). Valid, and far too conservative.
• H3d-ii refuted: the betting interval is no better than Bernstein for this purpose. Control-variate width over real-only width averages 3.15 at n = 30 and 2.89 at n = 60 (range 1.92 to 3.35), including all seven pairs with rho at least 0.85 at n = 60 (3c: 2.97 to 3.01). The low-rho half of the prediction holds (one pair). Real-only betting widths are 0.293 (n = 30) and 0.159 (n = 60), against a sample sd of 0.1 to 0.15.
• Why (measured). The paper's plug-in bet is capped at 1/2 over the candidate mean to keep capital positive on a [0, 1] range. With outcome sd near 0.1 the variance-optimal bet is far above the cap: the cap binds on 99 to 100% of bets at n = 60 (none-0: 100%, delay-3: 99%, 200 draws each). Capped bets grow capital at a rate set by the range, not the variance, so the interval's width is again range-limited, and changing the variance prior makes no difference (widths agree to three digits). The control-variate version pays range (1 + lam), a split error budget and a pilot third on top.
• Reading across 3c and 3d: any bound whose guarantee leans on the worst-case range [0, 1] hides the twin's variance reduction at n of 30 to 60. Two genuinely different routes remain, both untested: (b) a bound on the paired residual's range that is itself checked on a pilot (a bounded-discrepancy assumption, H3 as stated), and (c) a test that does not need an interval at all (a certification threshold test with a pre-registered discrepancy budget).
• Limitations: one betting construction (hedged capital, PrPl-EB); other bet schedules (e.g. cap 0.9 with a positivity guard, or mixture methods) may do better and were not tried; same single-task pools; 1,000 replications; union-bound split fixed at 50/50.
• Next: milestone 3e, a bounded-discrepancy certification test (route b/c) with the discrepancy bound estimated on a held-out pilot and its failure probability accounted for; then milestone 4.

2026-10-24 · Milestone 3c: a distribution-free interval (planned before running)

• Why: milestone 3b showed no resampling interval reaches nominal at n = 30 on bounded, skewed outcomes. A guarantee needs a construction that does not lean on normality. Question: what does a finite-sample-valid interval for the control-variate estimand cost in width, and does the twin still shorten it?
• Construction (experiments/m3c_dfree/dfree.py): outcomes lie in [0, 1], so R - lam T lies in [-lam, 1] for lam in [0, 1]. Split the n pairs: the first n1 = n/3 give lam (OLS slope clipped to [0, 1]); the other n2 = n - n1 give A = mean(R - lam T) with an empirical Bernstein bound (Maurer and Pontil 2009, two-sided, delta = 0.025, range 1 + lam); all N = n + m twin runs give B = mean(T) with an empirical Bernstein bound (delta = 0.025, range 1). Estimate A + lam B, half-width the sum of the two. The union bound makes it valid whatever the dependence between the parts. Real-only baseline: empirical Bernstein on all n real outcomes, delta = 0.05. Pool sampling is without replacement, which only tightens the bounds.
• H3b-i (validity): coverage of both intervals is at least 95% on every one of the 21 pairs at n = 30 and 60. Refuted if any pair falls below 95.0% (MC s.e. at most 0.5 points at 1,000 replications; coverage will be far above nominal, so a miss would be a bug or a wrong range).
• H3b-ii (value): the control-variate EB width over the real-only EB width is below 1 on every pair with paired correlation rho >= 0.85 at n = 60, and above 1 on pairs with rho < 0.3 (the twin costs width when it is uncorrelated). Refuted if either fails. Prior: the Bernstein terms are dominated by the range term at n = 30, so the gain may be small there; I expect the sign of the effect but not its size.
• Not tested: tuning delta between the two parts, a betting confidence sequence, a bounded-discrepancy certification test (H3 proper).

### Outcome (21 pairs, 1,000 replications per cell, n in {30, 60}; experiments/m3c_dfree/results/)

• H3b-i supported, trivially. Both intervals cover 100.0% on every pair and n (the minimum over all 42 cells). They are valid and far too conservative to be useful as stated.
• H3b-ii refuted, and the twin costs width. The control-variate EB interval is 2.1 to 3.1 times as wide as the real-only EB interval on every pair (mean 2.99 at n = 30, 2.89 at n = 60), including the seven pairs with rho at least 0.85 at n = 60 (ratios 2.97 to 3.01). The low-rho half of the prediction (ratio above 1 at rho < 0.3) holds, but only one pair has rho < 0.3 and the high-rho half fails everywhere.
• Why. The Bernstein bound is dominated by its range term, not its variance term: on none-0 at n = 60 the real-only range term is 0.146 against a variance term of 0.027 (sd of R is 0.078). The control-variate construction pays for a wider range (1 + lam), splits the error budget in two (delta 0.025 each), spends a third of the pairs on lam, and adds the bound on the twin mean; none of that is recovered by a variance reduction that the range term hides. So the 1/(1 - rho^2) gain of milestones 2 and 3 is real for the variance and invisible to a worst-case-range bound at these n.
• Consequence for the programme: a generic empirical-Bernstein interval is the wrong certification tool. The remaining routes are (a) a betting confidence sequence, whose width tracks the empirical variance rather than the range, (b) a bound that uses the observed range of the paired residual, and (c) a bounded-discrepancy assumption checked on the pilot (H3 as stated). Not run: any of them, and the choice of lam from a larger share of pairs.
• Limitations: same single-task pools; 1,000 replications; the bound is not tuned (delta split, pilot share fixed at one third).
• Next: a betting-style interval for the same estimand (route a), then milestone 4.

2026-10-24 · Milestone 3b: finite-sample intervals (planned before running)

• Why: H1 failed at milestone 3 because plug-in t intervals undercover on bounded, left-skewed outcomes (real-only itself reaches only 91.2% at n = 30). Question: does a resampling interval repair coverage for cv and ppi_pp, and at what width cost?
• Intervals compared, same pools and the same replication design as analyse.py (n real+twin pairs, m = 9n twin-only contexts, drawn without replacement from the 4,000-context pool; truth is the pool mean of R): t (existing plug-in), boot_pct (percentile bootstrap, pairs and extras resampled separately), boot_t (studentised bootstrap, plug-in standard error recomputed in every resample, lam re-estimated in every resample). B = 399, 2,000 replications per pair, n in {30, 60}, 21 pairs, methods cv and ppi_pp.
• H1b: the studentised bootstrap interval for cv has coverage of at least 93.0% on every one of the 21 pairs at n = 30. Refuted if any pair is below 93.0%. The Monte Carlo standard error is about 0.5 points at 2,000 replications, so pairs in 92.5 to 93.5 are reported as close calls. Prior: partial support; I expect it to repair most of the 1 to 4 point twin-specific loss but not all of the skew-driven loss at n = 30, so some payload-mass pairs may still miss.
• H1c: mean width of boot_t over t is at most 1.15 on every pair (the repair is not bought with intervals more than 15% wider). Refuted otherwise.
• Not tested here: a distribution-free bound (empirical Bernstein) for a fixed lam; with an estimated lam it is not valid as written, so it is left for the certification milestone.

### Outcome (21 pairs, 2,000 replications per cell, B = 399; experiments/m3b_intervals/results/; MC s.e. about 0.5 points)

• H1b refuted as pre-registered, but most of the gap closes. Studentised bootstrap for cv at n = 30: mean coverage 93.6% (plug-in t: 92.0%; percentile bootstrap: 90.7%), minimum 90.8% (payload-mass-3, the most skewed pair); 8 of 21 pairs are below 93.0 (13 are at or above), 17 of 21 are at or above 92.5, i.e. within one MC s.e. of the floor. For ppi_pp the minimum is 91.1% and 7 of 21 pairs are below 93.0. At n = 60 the cv minimum is 92.4% (3 of 21 below 93.0) and the ppi_pp minimum 92.95% (1 of 21).
• The percentile bootstrap is worse than the plug-in t (every pair below 93.0 at n = 30, mean 90.7% for cv): with 30 skewed observations it inherits the skew and the small-sample bias of resampling without correcting the studentisation. Only the studentised version helps.
• H1c refuted. The cost is width: boot_t over t averages 1.14 (cv) and 1.19 (ppi_pp) at n = 30 and reaches 1.30 and 1.33 on mass-3; at n = 60 the mean is 1.08 and 1.12. So the repair is real but not free, and the residual undercoverage sits on exactly the pairs with the skewest outcome, where a bootstrap-t is known to be weakest.
• Conclusion for the programme: use the studentised bootstrap as the working interval from n = 60; at n = 30 no resampling interval here reaches nominal on bounded, skewed outcomes. A guarantee needs a distribution-free construction (empirical Bernstein or a betting confidence sequence, with lam fixed or sample-split), which is the certification milestone, and it will be wider still. Not yet tested: BCa, or a transformation (logit of the bounded outcome) before the interval.
• Limitations: same pools as milestone 3 (one task, one outcome, truth is the pool mean of 4,000 contexts, resampling without replacement from that pool); model-versus-model pairs.
• Next: milestone 4 (SIMPLER paired tables), or a distribution-free interval for H3 first.

2026-10-24 · Milestone 3: benchmark v1, MuJoCo pairs (planned before running the estimators)

• Built experiments/m3_benchmark/pairs.py: a torque-driven pendulum arm in MuJoCo (RK4, 5 ms steps, 3 s) tracking a sinusoid with PD plus gravity feedforward. The twin is the nominal model; the "real" model adds an Ornstein-Uhlenbeck disturbance torque (so the paired correlation is below 1 even with no omitted physics) and one omitted effect (joint friction, command delay, sensor noise, actuator gain shortfall, payload-mass error) at one of four swept magnitudes, plus a no-effect pair: 21 pairs. Magnitudes are swept, not fitted, so these are model-versus-model pairs; no datasheet is claimed. Before the magnitudes were set, a 40-context probe gave a paired correlation of 0.999 for every pair, so the disturbance and magnitudes were raised until the pairs spread over roughly 0.25 to 0.94 (a change made to get an informative benchmark, looking only at the correlation and not at any estimator's coverage).
• Population: 4,000 contexts per pair, one twin run and one real run each; the truth for a pair is the mean real outcome over the 4,000.
• Pre-registered analysis (analyse.py, 4,000 replications per pair, n real+twin pairs and m = 9n extra twin-only contexts, drawn without replacement): - H1: coverage of the 95% intervals of cv and ppi_pp at n = 30. Refuted if either is below 93.0% (nominal minus 2 points) on any pair. The Monte Carlo standard error of a coverage figure is about 0.34 points, so a pair at 93.0 to 93.7 is a close call and is reported as such. ppi and real_only are reported for contrast. - H2: E = variance of real-only divided by variance of the estimator, over the replications. Paired uses the context-matched twin run; unpaired uses twin runs on independent contexts (same n, same extra runs, same estimator). Supported on a pair if the lower 95% bootstrap bound of E_paired / E_unpaired is at least 2; refuted if that interval includes 1 on most pairs. I expect pairs with rho below about 0.5 to give a ratio near 1, since the gain is 1/(1 - rho^2). - H6: across the 21 pairs, Spearman correlation of E (cv, n = 30) with (a) detectability, the Kolmogorov-Smirnov distance between the marginals of R and T, and (b) the paired correlation rho. H6 is supported if (b) exceeds (a) in absolute value with the bootstrap-over-pairs interval of the difference excluding 0. Refuted if detectability predicts E as well as rho does. Honest prior: E is a near-monotone function of rho by theory, so (b) should be large; whether (a) is small is the open part, since the omitted effects here shift both the mean and the correlation.

### Outcome (21 pairs, 4,000 contexts each, 4,000 replications per pair and n; experiments/m3_benchmark/results/)

• Sanity check against theory. Measured E of the control variate at n = 30, N = 300 is within 6% of the closed form 1/(1 - 0.9 rho^2) on every pair (mean 3.8% below, the cost of estimating lam). Paired correlations span 0.17 to 0.88.
• H1 refuted as pre-registered. At n = 30 the 95% intervals of cv and ppi_pp fall below 93.0% on 15 and 20 of 21 pairs (worst 88.8% and 86.7%, on the payload-mass pairs). The cause is not only the twin: the real-only t interval also undercovers (minimum 91.2%) because the outcome is bounded and strongly left-skewed (skewness -2.4 on the worst pairs), and the twin estimators lose a further 1 to 4 points on top of that. At n = 60 the minima are 92.4% (real-only), 91.0% (cv), 89.8% (ppi_pp), so the shortfall shrinks but remains. ppi (lam = 1) is not better. Conclusion: plug-in t intervals are not a valid basis for H1; the finite-sample interval the proposal calls for (milestone 3b) is needed.
• H2 supported. Paired E over unpaired E has a lower 95% bound of at least 2 on 13 of 21 pairs and an interval that includes 1 on none. It fails the 2x bar exactly where rho is below about 0.75 (E between 1.0 and 1.9). Unpaired twin runs never help: E is 0.96 to 0.98 on every pair, a loss of 2 to 4% from estimating lam on uncorrelated data.
• H6 mixed. Spearman(E, rho) = 0.999, but Spearman(E, KS detectability) = -0.954 and with the standardised mean shift -0.855. The interval of |rho| minus |KS| over pairs is [0.008, 0.146], so rho predicts value significantly better, yet detectability also predicts it well: the claim that it does not is refuted on this benchmark. Reason: every omitted effect here both shifts the outcome distribution and decorrelates the twin. The benchmark has no highly detectable, highly correlated pair (a pure offset), which is where H6 would be tested properly.
• Limitations: model-versus-model pairs with swept, not sourced, magnitudes; one task; one outcome; context-paired with a deterministic twin only.
• Next: milestone 3b, a finite-sample interval (studentised bootstrap and an empirical-Bernstein bound) re-tested on the same pools; add offset-type pairs for H6; then milestone 4 (SIMPLER).

2026-10-24 · Milestone 2 part 1: consolidated estimator library

• Hypothesis written before running: the three projects' closed forms are special cases of one family mean(R) - lam (mean(T_paired) - mean(T_all)), so one library with lam in {0, 1, power-tuned, OLS} should reproduce their numbers, and PPI (lam = 1) should lose to real-only when the twin's sensitivity differs from reality's (twin scale g = 2).
• Result: src/sim2real_inference/ (stdlib only) and 11 tests pass. All four estimators are unbiased against a twin bias of 0.7; the exact fixed-lam variance var_ppi equals var_cv at the optimal lam, and matches simulation within 7%; the 95% intervals of PPI, PPI++ and CV cover above 92% at n = 30, N = 300. With g = 2, PPI's variance exceeds real-only's while CV's is below it, which is the first concrete case for tuning lam (H1 groundwork, not yet a result on benchmark pairs).
• Caught on the way: a test I wrote used the noise-free correlation for a noisy twin and failed at variance ratio 1.50; the code was right, the test was wrong.
• Not done: the synthesis table of what the ~130 *-twin projects say about rho, and twin-pilot's two-stage procedure is not ported. Next: the synthesis, then milestone 3 (MuJoCo pairs).

2026-10-24 · Programme opened

• Wrote PROPOSAL.md: 13 references checked against arXiv and Crossref, six falsifiable hypotheses, seven milestones, target CoRL.
• Prior work in this repository: about 130 -twin, twin- and mimic-twin* projects, each a stylised model of one way a twin misleads. twin-evaluation and twin-pilot already cite prediction-powered inference and SIMPLER, so the novelty claimed here is narrower: unification under context pairing, the certification boundary, active and anytime-valid evaluation, and real paired data.
• Next: milestone 2. Build one estimator library that reproduces the analytic results of twin-evaluation, twin-pilot and twin-certification as tests, and write a synthesis table of what the twin projects found about the paired correlation ρ.