How far can DiLoCo reports be compressed?

What it found

Unbiased compression of each worker’s pseudo-gradient (noise proportional to the signal, ω) has exact stationary variance αV_w(1+ω)/(Ms(2−αsc)), c = 1+ω/M: it caps the outer step at 2/(sc) and the contraction at 1−1/c (keeping 10% of coordinates with 8 workers: 0.53, 22 rounds to 10⁻⁶ against 1), and random participation is the special case ω = 1/p−1. Dithered quantisation only adds Δ²/12 to the variance and leaves stability alone. At fixed bandwidth, sparsifying favours few full reports, while quantising has an interior optimum near 1.3 bits per coordinate with many workers, about 4× below 8-bit reports (the textbook high-resolution law understates the noise 2.9× at 1 bit). Matched to simulation within 3%. Stylised.

A figure from its own output

Paper, code and results

• README and method
• White paper
• Experiments and results
• Code
• Tests

More in Digital twins & sim-to-real

• What does it cost to send only a fraction of the outer update, and does error feedback help?Sending a fraction p of the outer update: unbiased rescaling costs 1/p in noise floor, error feedback costs only 1+αs(1/p−1) at small steps but caps contraction at √(1−p); exact closed form, stability limits, horizon losses.
• Can a DiLoCo-style run hide its synchronisation by applying the outer update rounds late?Overlapping the DiLoCo sync by applying the outer step τ rounds late does not pay: stable iff αs < 2 sin(π/(4τ+2)), best rate 1−2 sin(π/(4τ+2))/κ_H (exactly κ_H/(κ_H+1) at τ=1), momentum is useless once τ≥1 (69 → 1388 rounds at κ_H=100), hidden-sync gain ≤ (τ+1) sin(π/(4τ+2)) ≤ 1 (search: 1.5–4.9× slower than blocking averaging), fixed if the fresh pseudo-gradient has weight >½.
• A twin whose wind gusts are matched to the logged variance each step: how wrong is the position variance of a PD-held vehicle, and does matching the spectrum instead fix it?A twin whose wind gusts are iid per step with the logged variance: real position variance is s²(a+k_d)/(k_d k_p(a²+a k_d+k_p)), the twin's is 141× smaller at a=1, dt=0.01 and scales as dt, its 1e-3 margin is exceeded 78% of the time, and a spectrum-matched white twin is always conservative.
• A free-space radar twin ignores ground-bounce lobing: how wrong is it, and can a two-ray twin with a slightly wrong target height be worse?Free-space radar twin vs two-ray ground-bounce lobing: detection fraction exactly 2/3 at the free-space range edge, real detects 44% beyond it, a two-ray twin with height error δh disagrees ≈4h_rδh/(λr) and is worse than free space at short range, a height-jitter ensemble scores Brier 0.119 vs 0.197 deterministic. Stylised.
• A point-target radar twin of two unresolved scatterers: it has the right median, so what does it get wrong, and does averaging looks converge where it says?Point-target radar twin of two unresolved scatterers under ideal monopulse: real mean is exactly the strong scatterer and median exactly the power centroid, outside-span probability arccos(r)/π, a variance-matched Gaussian twin is 27× too optimistic at a 3σ gate (r=0.9), averaging looks floors at bias 2r²/(1+r²) where the twin predicts √(var/N), and the median beats the mean for r≥0.477. Stylised.
• A constant-variance Gaussian camera twin of Poisson photon counting: which dark/bright decision rule does it teach, and what does that cost on the real sensor?Constant-variance Gaussian camera twin of Poisson photon counting: the real Bayes threshold is the logarithmic mean, the twin's the arithmetic mean (7.28 vs 8.00 for 4 vs 12), a detector tuned in it has 1.25–1.83× the optimal error and 23× at λ0=200 (λ1=2λ0), and a square-root twin recovers the optimum in 8 of 9 pairs. Stylised.

All 164 Digital twins & sim-to-real projects →

Other research topics

Proper scoring & elicitation28Effort & peer prediction9Verification games & slashing19Detection & tolerances14Markets as routers13Prediction-market design9Paying contributors8Decentralised training systems23Digital twins & sim-to-real164LLM chip design loops40Robot manufacturing loops53Ambient AI assistant46Self-replicating manufacturing34Simulation-first design64Engineered clothing45Task-specific AI accelerators40Algorithmic chip-design loops42Open-source purpose-built robots44The software-engineering loop43Adversarial human imitation41Open-source residential construction41Autonomous multi-angle capture34Sensing, imaging and 3D31Energy supply and dense energy storage39Robot societies among people31Agentic co-design of low-cost robots27Laundry-hanging robots25Machine learning on qualitative data23Robots on the building site27Civilizations & group psychology1History & human sciences1

Produced with AI assistance (Claude). Results are from the code in this repository and are simulated or stylised unless the abstract says otherwise; nothing here implies affiliation with or endorsement by any lab, researcher or company cited. MIT licensed.