What happens to local SGD when workers drop in and out?

research/partial-participation

Local SGD when workers drop in and out: exact floors for participant averaging (E[1/K|K≥1]) and expected-count normalisation (stable iff αsc<2), at matched speed the fixed normaliser is quieter by E[K]E[1/K] at small steps, and reliable workers bias the fixed point unless inverse-propensity weighted.

11 tests

What it found

Each of N workers joins a round with probability p. Averaging the participants has exact stationary variance αV_wm/(s(2−αs)) with m = E[1/K | K≥1]; dividing by the expected count instead is stable only for αs < 2/c, c = 1+(1−p)/(Np), and cannot contract faster than 1−1/c per round (0.22 at N=32, p=0.1). At matched speed the fixed normaliser is quieter at small steps by E[K|K≥1]·E[1/K|K≥1] (1.34×), and averaging wins only at fast rates. Reliable workers bias the fixed point (0.467 instead of 1.000 in our example) unless displacements are inverse-propensity weighted. Matched to simulation within 0.5%. Stylised.

A figure from its own output

Figure produced by research/partial-participation: matched_speed — partial-participation/experiments/matched_speed.svg, written by the project's own experiment code.

Paper, code and results

More in Decentralised training systems

What does compressing the pseudo-gradient cost local SGD, and is it better than syncing less often?Local SGD with compressed displacements: exact floor (1+ω) and stability limit αs(1+ω/N)<2 for rand-k (more workers restore the step, not the floor), closed form for norm-scaled quantisation, and at equal bytes syncing less often beats compressing.
How much outer step does overlapping communication with compute cost, and how many rounds should be in flight?Overlapping communication in local SGD with τ-round-late outer updates: exact stable step 2 sin(π/(4τ+2)) (shrinks ~1/τ), double-root rate τ/(τ+1), exact noise floor matched to simulation, free at small steps, wall-clock-optimal delay ⌈L/T_c⌉ (1.9–7.5× faster than no overlap).
Does error feedback rescue the noise floor that compressed local SGD loses?Error feedback in compressed local SGD: exact stability limit 2(2−ρ)Nρ/(Nρ²+4(1−ρ)) and closed-form floor that returns to the uncompressed one at small steps (unbiased compression pays 1/ρ), dropping is biased to the send-rate-weighted mean and the residual removes it at a variance price.
How should a server weight workers that ran different numbers of inner steps?Weighting workers that ran different numbers of inner steps: exact floor α²Σw²V/(r(2−r)), worker information I∞·tanh(λH/2) saturates, optimal weights span ≤2× so equal weights lose ≤9/8 (measured ≤1.09) while step-proportional weights lose up to 1.76× per mode and 10.5× on three modes.
What does a median or trimmed-mean outer step cost local SGD, and what does it buy against Byzantine workers?Median and trimmed-mean outer aggregation in local SGD: exact linear loop with floor and bias split, π/2 variance price with no attackers, bounded worst-case bias Φ(z)=1/(2(1−ε)) that more workers cannot average away but a longer inner loop shrinks.
Which of sync interval, compression, delay, momentum and robust aggregation sets the final loss of low-communication training?Research programme (proposal stage): a predictive theory of DiLoCo-style low-communication training, testing on real small transformers the closed forms derived in about a dozen quadratic-model projects, with 16 references, hypotheses H1 to H6 and an ICML/NeurIPS workshop as the first target

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.