How much gradient noise survives in DiLoCo-style local SGD, and how should the outer step, momentum, worker count and sync interval be set against it?

research/noisy-local-sgd

Noise floor of DiLoCo-style local SGD: exact stationary loss αV(1+β)/((1−β)s(2(1+β)−αs)), independent of H at α=1, momentum free at equal effective step, near-linear speedup in M, and a noise-limited sync interval that grows as the floor target falls.

9 tests

What it found

On a quadratic with M workers, H noisy inner steps and a heavy-ball outer step, the stationary variance per mode is exactly αV(1+β)/((1−β)s(2(1+β)−αs)) with V the averaged inner noise and s = 1 − (1 − ηa)^H, matching literal simulation to 0.2%. At α = 1 the floor is independent of H; for small α more local steps shrink it by less than a factor of 2. Momentum costs no extra floor at equal effective step (and lowers it at large steps), workers give near-linear speedup (331× at M = 256, α still below 1), and unlike the noise-free case the best sync interval grows as the noise-floor target falls (H* = 5, 11, 38 at C = 10), up to 41× faster than syncing every step. Stylised: quadratic, Gaussian noise, shared Hessian, fixed α.

A figure from its own output

Figure produced by research/noisy-local-sgd: floor_vs_h — noisy-local-sgd/experiments/floor_vs_h.svg, written by the project's own experiment code.

Paper, code and results

More in Decentralised training systems

What happens to local SGD when workers drop in and out?Local SGD when workers drop in and out: exact floors for participant averaging (E[1/K|K≥1]) and expected-count normalisation (stable iff αsc<2), at matched speed the fixed normaliser is quieter by E[K]E[1/K] at small steps, and reliable workers bias the fixed point unless inverse-propensity weighted.
What does compressing the pseudo-gradient cost local SGD, and is it better than syncing less often?Local SGD with compressed displacements: exact floor (1+ω) and stability limit αs(1+ω/N)<2 for rand-k (more workers restore the step, not the floor), closed form for norm-scaled quantisation, and at equal bytes syncing less often beats compressing.
How much outer step does overlapping communication with compute cost, and how many rounds should be in flight?Overlapping communication in local SGD with τ-round-late outer updates: exact stable step 2 sin(π/(4τ+2)) (shrinks ~1/τ), double-root rate τ/(τ+1), exact noise floor matched to simulation, free at small steps, wall-clock-optimal delay ⌈L/T_c⌉ (1.9–7.5× faster than no overlap).
Does error feedback rescue the noise floor that compressed local SGD loses?Error feedback in compressed local SGD: exact stability limit 2(2−ρ)Nρ/(Nρ²+4(1−ρ)) and closed-form floor that returns to the uncompressed one at small steps (unbiased compression pays 1/ρ), dropping is biased to the send-rate-weighted mean and the residual removes it at a variance price.
How should a server weight workers that ran different numbers of inner steps?Weighting workers that ran different numbers of inner steps: exact floor α²Σw²V/(r(2−r)), worker information I∞·tanh(λH/2) saturates, optimal weights span ≤2× so equal weights lose ≤9/8 (measured ≤1.09) while step-proportional weights lose up to 1.76× per mode and 10.5× on three modes.
What does a median or trimmed-mean outer step cost local SGD, and what does it buy against Byzantine workers?Median and trimmed-mean outer aggregation in local SGD: exact linear loop with floor and bias split, π/2 variance price with no attackers, bounded worst-case bias Φ(z)=1/(2(1−ε)) that more workers cannot average away but a longer inner loop shrinks.

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.