How should the outer optimiser and sync interval be set when workers run H local steps per round?

What it found

On a quadratic, H inner steps make the outer optimiser see curvatures 1 − (1 − ηa)^H, which saturate at 1: the outer condition number shrinks from κ to about κ/H (1000 → 16.1 at H=64). Per-round rates are exact 2×2 spectral radii (matching literal simulation), the tuned heavy-ball outer step puts every mode at exactly √β*, and the best momentum falls from 0.88 at H=1 to 0.01 at H=1024, so a fixed Nesterov 0.7/0.9 needs 4–8× more rounds. With a sync costing C inner steps, the wallclock-optimal interval is H* ≈ C, worth about √C/2 over syncing every step (16× at C=1000), while plain averaging has no interior optimum. Stylised: noiseless quadratic with a shared Hessian.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Decentralised training systems

• How much gradient noise survives in DiLoCo-style local SGD, and how should the outer step, momentum, worker count and sync interval be set against it?Noise floor of DiLoCo-style local SGD: exact stationary loss αV(1+β)/((1−β)s(2(1+β)−αs)), independent of H at α=1, momentum free at equal effective step, near-linear speedup in M, and a noise-limited sync interval that grows as the floor target falls.
• What happens to local SGD when workers drop in and out?Local SGD when workers drop in and out: exact floors for participant averaging (E[1/K|K≥1]) and expected-count normalisation (stable iff αsc<2), at matched speed the fixed normaliser is quieter by E[K]E[1/K] at small steps, and reliable workers bias the fixed point unless inverse-propensity weighted.
• What does compressing the pseudo-gradient cost local SGD, and is it better than syncing less often?Local SGD with compressed displacements: exact floor (1+ω) and stability limit αs(1+ω/N)<2 for rand-k (more workers restore the step, not the floor), closed form for norm-scaled quantisation, and at equal bytes syncing less often beats compressing.
• How much outer step does overlapping communication with compute cost, and how many rounds should be in flight?Overlapping communication in local SGD with τ-round-late outer updates: exact stable step 2 sin(π/(4τ+2)) (shrinks ~1/τ), double-root rate τ/(τ+1), exact noise floor matched to simulation, free at small steps, wall-clock-optimal delay ⌈L/T_c⌉ (1.9–7.5× faster than no overlap).
• Does error feedback rescue the noise floor that compressed local SGD loses?Error feedback in compressed local SGD: exact stability limit 2(2−ρ)Nρ/(Nρ²+4(1−ρ)) and closed-form floor that returns to the uncompressed one at small steps (unbiased compression pays 1/ρ), dropping is biased to the send-rate-weighted mean and the residual removes it at a variance price.
• How should a server weight workers that ran different numbers of inner steps?Weighting workers that ran different numbers of inner steps: exact floor α²Σw²V/(r(2−r)), worker information I∞·tanh(λH/2) saturates, optimal weights span ≤2× so equal weights lose ≤9/8 (measured ≤1.09) while step-proportional weights lose up to 1.76× per mode and 10.5× on three modes.

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.