Does error feedback rescue the noise floor that compressed local SGD loses?

research/error-feedback-sync

Error feedback in compressed local SGD: exact stability limit 2(2−ρ)Nρ/(Nρ²+4(1−ρ)) and closed-form floor that returns to the uncompressed one at small steps (unbiased compression pays 1/ρ), dropping is biased to the send-rate-weighted mean and the residual removes it at a variance price.

9 tests

What it found

Workers keep the unsent part of their displacement and add it to the next message. On a quadratic with random sparsification the second moments close exactly: stability requires αs < 2(2−ρ)Nρ/(Nρ²+4(1−ρ)) — half the unbiased limit for a single worker, above the uncompressed limit for large cohorts — and the floor has a closed form that returns to the uncompressed one at small steps (1.08× at ρ=¼ where unbiased compression pays 4×) while the mean keeps its speed. What the residual buys is unbiasedness: dropping converges to the send-rate-weighted mean, feedback to the true mean, but at a variance that only pays off for very small steps. Top-k is simulation only; stylised quadratics.

A figure from its own output

Figure produced by research/error-feedback-sync: stability_limit — error-feedback-sync/experiments/stability_limit.svg, written by the project's own experiment code.

Paper, code and results

More in Decentralised training systems

How should a server weight workers that ran different numbers of inner steps?Weighting workers that ran different numbers of inner steps: exact floor α²Σw²V/(r(2−r)), worker information I∞·tanh(λH/2) saturates, optimal weights span ≤2× so equal weights lose ≤9/8 (measured ≤1.09) while step-proportional weights lose up to 1.76× per mode and 10.5× on three modes.
What does a median or trimmed-mean outer step cost local SGD, and what does it buy against Byzantine workers?Median and trimmed-mean outer aggregation in local SGD: exact linear loop with floor and bias split, π/2 variance price with no attackers, bounded worst-case bias Φ(z)=1/(2(1−ε)) that more workers cannot average away but a longer inner loop shrinks.
Which of sync interval, compression, delay, momentum and robust aggregation sets the final loss of low-communication training?Research programme (proposal stage): a predictive theory of DiLoCo-style low-communication training, testing on real small transformers the closed forms derived in about a dozen quadratic-model projects, with 16 references, hypotheses H1 to H6 and an ICML/NeurIPS workshop as the first target
How far over its limit can a rate limiter be pushed in a true rolling window?How far over its limit can a rate limiter be pushed in a true rolling window? Adversarial search over fixed window, sliding log, sliding-window counter, token bucket and leaky bucket: only the sliding log holds the limit; the sliding counter admits up to about 1.5 L and the buckets nearly 2 L, and under Poisson overload at 1.5x load 39-47% of rolling windows exceed the limit for fixed window and buckets. Stylised.
What can one stubborn node do to gossip training?Pairwise gossip for decentralised training: exact contraction (n−2)/(2(n−1))≈½ per matching round (~20 rounds at any n), noise floor (n−1)(n−2)s²/n, one stubborn node captures consensus at rate 1−1/(n(n−1)), trust-region clipping caps drift at τ/(n(n−1)) per step.
What does infrequent averaging do to the answer?Local-SGD/DiLoCo-style averaging on heterogeneous quadratics: exact fixed point with saturating weights 1-(1-ηa_i)^H, non-monotone bias to the unweighted mean, exact a_i/w_i correction, step-count inflation moves the fixed point 0.50→0.91, wall-clock-optimal sync interval.

All 23 Decentralised training systems projects →

Other research topics

Proper scoring & elicitation28 Effort & peer prediction9 Verification games & slashing19 Detection & tolerances14 Markets as routers13 Prediction-market design9 Paying contributors8 Decentralised training systems23 Digital twins & sim-to-real164 LLM chip design loops40 Robot manufacturing loops53 Ambient AI assistant46 Self-replicating manufacturing34 Simulation-first design64 Engineered clothing45 Task-specific AI accelerators40 Algorithmic chip-design loops42 Open-source purpose-built robots44 The software-engineering loop43 Adversarial human imitation41 Open-source residential construction41 Autonomous multi-angle capture34 Sensing, imaging and 3D31 Energy supply and dense energy storage39 Robot societies among people31 Agentic co-design of low-cost robots27 Laundry-hanging robots25 Machine learning on qualitative data23 Robots on the building site27 Civilizations & group psychology1 History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.