How fast does a shared model forget a task, and does splitting it into modules help?

research/forgetting-law

Exact catastrophic-forgetting law for sequential training on random low-rank tasks: (r/d)(1−r/d)(ρᵏ−λᵏ) rises then vanishes, uniform module routing dilutes it but sharing wins on seen-plus-fresh loss, conflicting tasks leave a floor 2rτ²/d.

9 tests

What it found

Training to convergence on random rank-r tasks in R^d is a projection, and the expected loss on a task learned k tasks ago is exactly (r/d)(1−r/d)(ρ^k−λ^k): zero when learned, a peak (0.044 at d=12, r=3, lag 2.5) at a lag near 0.7d/r, then decay because the tasks agree; matched by simulation to the third decimal. Routing tasks to m modules dilutes both rates by m and cuts short-horizon forgetting (5.9× at T=4, m=8) but a single shared model minimises seen-plus-fresh loss at every horizon tried, and two modules forget more than one at T=12. With conflicting tasks forgetting has a floor 2rτ²/d that only task-specific modules remove. Stylised: linear regression, exact convergence, isotropic tasks.

A figure from its own output

Figure produced by research/forgetting-law: forgetting — forgetting-law/experiments/forgetting.svg, written by the project's own experiment code.

Paper, code and results

More in Decentralised training systems

How should independently trained modules be merged, and how much does parallel training buy?Merging independently trained modules on random low-rank tasks: exact residual 1−2amp+a²(mp+m(m−1)p²), optimal scale 1/(1+(m−1)p) (averaging stalls at (1−p)², unscaled sum diverges), harmonic 1/m decay, parallel speedup only logarithmic (8.9× at m=16, d/r=8).
When should a continual learner replay an old task, and does a fixed schedule beat random replay?Exact expected forgetting under replay schedules in sequential training on random low-rank tasks: 2×2 matrix recursion matches simulation, one replay is best at the forgetting-peak lag yet removes only 25–42%, a fixed period beats random replay only below ~1.2–3 peak lags, halving forgetting needs rate 0.25 (d=12, r=3) to 0.015 (d=128, r=1).
How much of an old task must a continual learner store, and is it better to replay a little often or a lot rarely?Partial replay when only m of an old task's r examples are stored: exact 2×2 law interpolating no replay and full replay, best-step saving nearly linear in m/r, and at a fixed example budget a few large replays beat many small ones by only 0.1–6 points.
When continual-learning tasks conflict, how much does a task-aware router of modules actually buy?Task-aware routing when continual-learning tasks conflict: the long-run floor is (2r/d)[τ_w²+τ_b²(1−Purity)] with purity = 1 − the router's Gini impurity, random modules buy nothing, a perfect router removes (m−1)/(K−1) of the between-cluster floor, and a 10% routing error gives back 36%.
How should the outer optimiser and sync interval be set when workers run H local steps per round?DiLoCo-style outer momentum: H inner steps precondition the outer quadratic (κ_H ≈ κ/H), exact per-mode rates, optimal momentum falls 0.88 → 0.01 with H (fixed 0.7/0.9 Nesterov needs 4–8× more rounds), and the wallclock-optimal sync interval is H* ≈ C (sync cost in inner steps), worth ≈ √C/2.
How much gradient noise survives in DiLoCo-style local SGD, and how should the outer step, momentum, worker count and sync interval be set against it?Noise floor of DiLoCo-style local SGD: exact stationary loss αV(1+β)/((1−β)s(2(1+β)−αs)), independent of H at α=1, momentum free at equal effective step, near-linear speedup in M, and a noise-limited sync interval that grows as the floor target falls.

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.