How stale can a shared rollout be before it is worthless?

research/stale-rollouts

Stale off-policy rollouts in swarm RL: exact importance-weight moments (E w²=e^{δ²}, ESS fraction e^{−δ²}), closed-form truncation bias/variance verified by simulation, best cap cuts shift-estimation MSE 1.9× at δ=1 and ~750× at δ=3, and the staleness window √ln(1/ρ)/drift.

7 tests

What it found

With Gaussian policies whose mean has moved by δ standard deviations, the importance weight has exact moments E w = 1 and E w² = e^δ², so the effective-sample fraction is e^−δ² (0.37 at δ=1) and a learner needing fraction ρ accepts a lag of √ln(1/ρ)/drift policy versions. Truncating weights has closed-form lost mass, bias and variance (matched by simulation); the best cap cuts the mean-shift estimator’s MSE 1.9× at δ=1, 12.8× at δ=2 and about 750× at δ=3, and finite-sample ESS is optimistic once weights are heavy-tailed. Equal-variance Gaussians, one statistic.

A figure from its own output

Figure produced by research/stale-rollouts: ess_vs_lag — stale-rollouts/experiments/ess_vs_lag.svg, written by the project's own experiment code.

Paper, code and results

More in Decentralised training systems

How fast does a shared model forget a task, and does splitting it into modules help?Exact catastrophic-forgetting law for sequential training on random low-rank tasks: (r/d)(1−r/d)(ρᵏ−λᵏ) rises then vanishes, uniform module routing dilutes it but sharing wins on seen-plus-fresh loss, conflicting tasks leave a floor 2rτ²/d.
How should independently trained modules be merged, and how much does parallel training buy?Merging independently trained modules on random low-rank tasks: exact residual 1−2amp+a²(mp+m(m−1)p²), optimal scale 1/(1+(m−1)p) (averaging stalls at (1−p)², unscaled sum diverges), harmonic 1/m decay, parallel speedup only logarithmic (8.9× at m=16, d/r=8).
When should a continual learner replay an old task, and does a fixed schedule beat random replay?Exact expected forgetting under replay schedules in sequential training on random low-rank tasks: 2×2 matrix recursion matches simulation, one replay is best at the forgetting-peak lag yet removes only 25–42%, a fixed period beats random replay only below ~1.2–3 peak lags, halving forgetting needs rate 0.25 (d=12, r=3) to 0.015 (d=128, r=1).
How much of an old task must a continual learner store, and is it better to replay a little often or a lot rarely?Partial replay when only m of an old task's r examples are stored: exact 2×2 law interpolating no replay and full replay, best-step saving nearly linear in m/r, and at a fixed example budget a few large replays beat many small ones by only 0.1–6 points.
When continual-learning tasks conflict, how much does a task-aware router of modules actually buy?Task-aware routing when continual-learning tasks conflict: the long-run floor is (2r/d)[τ_w²+τ_b²(1−Purity)] with purity = 1 − the router's Gini impurity, random modules buy nothing, a perfect router removes (m−1)/(K−1) of the between-cluster floor, and a 10% routing error gives back 36%.
How should the outer optimiser and sync interval be set when workers run H local steps per round?DiLoCo-style outer momentum: H inner steps precondition the outer quadratic (κ_H ≈ κ/H), exact per-mode rates, optimal momentum falls 0.88 → 0.01 with H (fixed 0.7/0.9 Nesterov needs 4–8× more rounds), and the wallclock-optimal sync interval is H* ≈ C (sync cost in inner steps), worth ≈ √C/2.

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.