How should independently trained modules be merged, and how much does parallel training buy?

What it found

Averaging m modules trained to convergence on random rank-r tasks has an exact residual error 1−2amp+a²(mp+m(m−1)p²) with p=r/d, needing only isotropy. The optimal scale is 1/(1+(m−1)p) and leaves (1−p)/(1+(m−1)p): plain averaging stalls at (1−p)² (0.78 vs 0.47 at m=8, d/r=8) and unscaled task arithmetic diverges (56 at m=64). The decay is harmonic, not geometric, so m parallel workers give a wall-clock speedup that grows only logarithmically (8.9× at m=16, 17× at m=64) and halves in efficiency near m≈d/r×2.5. Seen and fresh losses are closed forms matched by simulation. Stylised: linear regression, exact convergence, consistent isotropic tasks.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Decentralised training systems

• When should a continual learner replay an old task, and does a fixed schedule beat random replay?Exact expected forgetting under replay schedules in sequential training on random low-rank tasks: 2×2 matrix recursion matches simulation, one replay is best at the forgetting-peak lag yet removes only 25–42%, a fixed period beats random replay only below ~1.2–3 peak lags, halving forgetting needs rate 0.25 (d=12, r=3) to 0.015 (d=128, r=1).
• How much of an old task must a continual learner store, and is it better to replay a little often or a lot rarely?Partial replay when only m of an old task's r examples are stored: exact 2×2 law interpolating no replay and full replay, best-step saving nearly linear in m/r, and at a fixed example budget a few large replays beat many small ones by only 0.1–6 points.
• When continual-learning tasks conflict, how much does a task-aware router of modules actually buy?Task-aware routing when continual-learning tasks conflict: the long-run floor is (2r/d)[τ_w²+τ_b²(1−Purity)] with purity = 1 − the router's Gini impurity, random modules buy nothing, a perfect router removes (m−1)/(K−1) of the between-cluster floor, and a 10% routing error gives back 36%.
• How should the outer optimiser and sync interval be set when workers run H local steps per round?DiLoCo-style outer momentum: H inner steps precondition the outer quadratic (κ_H ≈ κ/H), exact per-mode rates, optimal momentum falls 0.88 → 0.01 with H (fixed 0.7/0.9 Nesterov needs 4–8× more rounds), and the wallclock-optimal sync interval is H* ≈ C (sync cost in inner steps), worth ≈ √C/2.
• How much gradient noise survives in DiLoCo-style local SGD, and how should the outer step, momentum, worker count and sync interval be set against it?Noise floor of DiLoCo-style local SGD: exact stationary loss αV(1+β)/((1−β)s(2(1+β)−αs)), independent of H at α=1, momentum free at equal effective step, near-linear speedup in M, and a noise-limited sync interval that grows as the floor target falls.
• What happens to local SGD when workers drop in and out?Local SGD when workers drop in and out: exact floors for participant averaging (E[1/K|K≥1]) and expected-count normalisation (stable iff αsc<2), at matched speed the fixed normaliser is quieter by E[K]E[1/K] at small steps, and reliable workers bias the fixed point unless inverse-propensity weighted.

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.