How should a server weight workers that ran different numbers of inner steps?

research/heterogeneous-steps

Weighting workers that ran different numbers of inner steps: exact floor α²Σw²V/(r(2−r)), worker information I∞·tanh(λH/2) saturates, optimal weights span ≤2× so equal weights lose ≤9/8 (measured ≤1.09) while step-proportional weights lose up to 1.76× per mode and 10.5× on three modes.

9 tests

What it found

With worker i running H_i steps, weighted averaging has exact floor α²Σw_i²V_i/(r(2−r)), r = αΣw_is_i. One worker's information is I_∞tanh(λH/2): H=8 carries 20% of the ceiling and H=64 93%, so a fast device is worth far less than its step count. Information-optimal weights span at most 2×, hence equal weights lose at most 9/8 (measured ≤1.09), whereas step-proportional (FedAvg-style) weights lose up to 1.76× per mode and 10.5× on three modes when one worker in eight is 256× faster. Matched to simulation within 1%. Stylised: quadratic, shared optimum (no client drift), fixed H_i.

A figure from its own output

Figure produced by research/heterogeneous-steps: information — heterogeneous-steps/experiments/information.svg, written by the project's own experiment code.

Paper, code and results

More in Decentralised training systems

What does a median or trimmed-mean outer step cost local SGD, and what does it buy against Byzantine workers?Median and trimmed-mean outer aggregation in local SGD: exact linear loop with floor and bias split, π/2 variance price with no attackers, bounded worst-case bias Φ(z)=1/(2(1−ε)) that more workers cannot average away but a longer inner loop shrinks.
Which of sync interval, compression, delay, momentum and robust aggregation sets the final loss of low-communication training?Research programme (proposal stage): a predictive theory of DiLoCo-style low-communication training, testing on real small transformers the closed forms derived in about a dozen quadratic-model projects, with 16 references, hypotheses H1 to H6 and an ICML/NeurIPS workshop as the first target
How far over its limit can a rate limiter be pushed in a true rolling window?How far over its limit can a rate limiter be pushed in a true rolling window? Adversarial search over fixed window, sliding log, sliding-window counter, token bucket and leaky bucket: only the sliding log holds the limit; the sliding counter admits up to about 1.5 L and the buckets nearly 2 L, and under Poisson overload at 1.5x load 39-47% of rolling windows exceed the limit for fixed window and buckets. Stylised.
What can one stubborn node do to gossip training?Pairwise gossip for decentralised training: exact contraction (n−2)/(2(n−1))≈½ per matching round (~20 rounds at any n), noise floor (n−1)(n−2)s²/n, one stubborn node captures consensus at rate 1−1/(n(n−1)), trust-region clipping caps drift at τ/(n(n−1)) per step.
What does infrequent averaging do to the answer?Local-SGD/DiLoCo-style averaging on heterogeneous quadratics: exact fixed point with saturating weights 1-(1-ηa_i)^H, non-monotone bias to the unweighted mean, exact a_i/w_i correction, step-count inflation moves the fixed point 0.50→0.91, wall-clock-optimal sync interval.
How often should preemptible workers checkpoint?Checkpoint interval for preemptible decentralised training: exact Lambert-W optimum independent of restart cost, Young/Daly overshoot 5–25%, flat within 2×, synchronous efficiency wall (50% at n≈691), flaky workers cut throughput 5×.

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.