What does infrequent averaging do to the answer?

research/local-sgd-bias

Local-SGD/DiLoCo-style averaging on heterogeneous quadratics: exact fixed point with saturating weights 1-(1-ηa_i)^H, non-monotone bias to the unweighted mean, exact a_i/w_i correction, step-count inflation moves the fixed point 0.50→0.91, wall-clock-optimal sync interval.

8 tests

What it found

Local-SGD / DiLoCo-style averaging of heterogeneous quadratic workers has a closed-form fixed point: a weighted mean of the workers' optima with weights 1−(1−ηa_i)^H that saturate, so it is exactly unbiased at H=1, forgets curvature as H grows (converging to the unweighted mean), and is non-monotone in between. Weighting by a_i/w_i removes the bias exactly; if the server trusts claimed step counts, a low-curvature worker moves the fixed point from 0.50 to 0.91 by claiming 100 steps instead of 5, tying communication savings to verified work. Wall-clock-optimal H under a bias tolerance is 5.7× faster than H=1 at moderate comm cost, with a non-interval feasible set. Noise formulas match Monte Carlo. Stylised.

A figure from its own output

Figure produced by research/local-sgd-bias: bias_vs_h — local-sgd-bias/experiments/bias_vs_h.svg, written by the project's own experiment code.

Paper, code and results

More in Decentralised training systems

How often should preemptible workers checkpoint?Checkpoint interval for preemptible decentralised training: exact Lambert-W optimum independent of restart cost, Young/Daly overshoot 5–25%, flat within 2×, synchronous efficiency wall (50% at n≈691), flaky workers cut throughput 5×.
Should a pipeline replicate stages or skip them?Replicate or skip? pipeline stages on unreliable workers: exact binomial drop law, replicas cannot beat the shared-domain floor (1−ρ)^L, a 4-stage skip budget halves workers for a 1e-3 drop rate (240 → 96), cost-optimal design shifts from skipping to replication as skips get costly.
How should a model be cut across devices of unequal speed?Cutting a model across heterogeneous devices: exact greedy bottleneck partition (matches DP, with memory caps), W/V ≤ T* < W/V + m·w_max/V, speed-blind splitting costs exactly v̄/v_min (2.4× at σ=0.5), device order is the residual decision (sorts no better than arbitrary, swap search within 1.1%), memory inverse to speed costs 1.7×.
What does it cost to skip the slowest workers?Waiting for the fastest k of n workers: exact order-statistic round times (Pareto mean finite iff n−k+1 > 1/α), exact SGD time-to-accuracy with a minimum k and an interior optimum (34× over wait-for-all at s=0, 1.12× at s=32), joint step tuning, and the exact selection bias of the speed filter with Horvitz–Thompson correction.
Does a bigger swarm learn more from shared rollouts?Rollout sharing in swarm RL as a public-goods game: Nash gives every agent a lone agent's learning level whatever N (total effort caps at (a/c−1)/θ) while efficient welfare is 4.5× at N=256; at θ=1 one agent generates; a Pigouvian matching subsidy (fraction θ(N−1)/(1+θ(N−1)) of cost) implements the optimum; junk beats honesty iff c−ε > p(τ+F).
How stale can a shared rollout be before it is worthless?Stale off-policy rollouts in swarm RL: exact importance-weight moments (E w²=e^{δ²}, ESS fraction e^{−δ²}), closed-form truncation bias/variance verified by simulation, best cap cuts shift-estimation MSE 1.9× at δ=1 and ~750× at δ=3, and the staleness window √ln(1/ρ)/drift.

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.