What can one stubborn node do to gossip training?

What it found

Pairwise gossip averaging as a stand-in for all-reduce: disagreement contracts by exactly (n−2)/(2(n−1)) ≈ ½ per matching round (about 20 rounds at any n), the noise floor is (n−1)(n−2)s²/n (matched to 0.3%), a single stubborn node captures the honest mean at exact rate 1−1/(n(n−1)), and a trust-region clip turns capture into linear creep of at most τ/(n(n−1)) per step. Stylised.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Decentralised training systems

• What does infrequent averaging do to the answer?Local-SGD/DiLoCo-style averaging on heterogeneous quadratics: exact fixed point with saturating weights 1-(1-ηa_i)^H, non-monotone bias to the unweighted mean, exact a_i/w_i correction, step-count inflation moves the fixed point 0.50→0.91, wall-clock-optimal sync interval.
• How often should preemptible workers checkpoint?Checkpoint interval for preemptible decentralised training: exact Lambert-W optimum independent of restart cost, Young/Daly overshoot 5–25%, flat within 2×, synchronous efficiency wall (50% at n≈691), flaky workers cut throughput 5×.
• Should a pipeline replicate stages or skip them?Replicate or skip? pipeline stages on unreliable workers: exact binomial drop law, replicas cannot beat the shared-domain floor (1−ρ)^L, a 4-stage skip budget halves workers for a 1e-3 drop rate (240 → 96), cost-optimal design shifts from skipping to replication as skips get costly.
• How should a model be cut across devices of unequal speed?Cutting a model across heterogeneous devices: exact greedy bottleneck partition (matches DP, with memory caps), W/V ≤ T* < W/V + m·w_max/V, speed-blind splitting costs exactly v̄/v_min (2.4× at σ=0.5), device order is the residual decision (sorts no better than arbitrary, swap search within 1.1%), memory inverse to speed costs 1.7×.
• What does it cost to skip the slowest workers?Waiting for the fastest k of n workers: exact order-statistic round times (Pareto mean finite iff n−k+1 > 1/α), exact SGD time-to-accuracy with a minimum k and an interior optimum (34× over wait-for-all at s=0, 1.12× at s=32), joint step tuning, and the exact selection bias of the speed filter with Horvitz–Thompson correction.
• Does a bigger swarm learn more from shared rollouts?Rollout sharing in swarm RL as a public-goods game: Nash gives every agent a lone agent's learning level whatever N (total effort caps at (a/c−1)/θ) while efficient welfare is 4.5× at N=256; at θ=1 one agent generates; a Pigouvian matching subsidy (fraction θ(N−1)/(1+θ(N−1)) of cost) implements the optimum; junk beats honesty iff c−ε > p(τ+F).

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.