-
Eight seeds at H = 32: DiLoCo with keep-1/4 compression edges below uncompressed, still short of 2 se
With a dense last sync, rand-k keep 1/4 ends 0.007 nats below uncompressed DiLoCo at H = 32 over eight seeds (se 0.004), 7 of 8 seeds below zero.
-
DiLoCo compression at H = 32: between the H = 16 tie and the H = 64 gain, and within noise of zero
With a dense last sync, rand-k keep 1/4 ends 0.006 nats below uncompressed DiLoCo at H = 32 (se 0.008): between H = 16's tie and H = 64's 0.018 gain.
-
DiLoCo's compression gain needs long inner loops: at H = 16, keep 1/4 is free but no better
With a dense last sync, rand-k keep 1/4 beats uncompressed DiLoCo by 0.018 nats at H = 64, but at H = 16 it only ties (-0.0004, se 0.0035); keep 1/8 costs.
-
How much should DiLoCo compress its syncs? A sweet spot at keep 1/4, not at the gentlest setting
With a dense last sync, rand-k at keep 1/4 ends 0.018 nats below uncompressed DiLoCo at H = 64; keep 1/2 gives about half that, keep 3/4's noise none.
-
DiLoCo's clean-last-sync gain is noise, not sparsity, and it is biggest at the lightest compression
Rand-k at keep 1/4 on every sync but the last beats uncompressed DiLoCo by 0.018 nats; Gaussian noise does the same as rand-k; keep 1/16 stops paying.
-
Why DiLoCo compression looks free at long sync intervals: a training gain cancels the last round's cost
Forking 16 runs at their last sync shows rand-k noise helps training at H = 64 (-0.014 nats) while the final round costs +0.012. The two cancel.
-
Is DiLoCo's compression gain at long sync intervals an untuned baseline? Plain averaging says no
Outer lr 0.7, 1.5, 2 and Nesterov all train worse than plain averaging at H = 64, and rand-k with a clean last sync still beats it by 0.014 nats.
-
Do long DiLoCo inner loops wash out compression noise? No, they keep it
One round of rand-k noise injected mid-training keeps all its parameter energy over 256 inner steps, and costs more loss at H = 64 than at H = 16. 8 forked runs
-
Is rand-k's sparsity why DiLoCo compression is free at long sync intervals? No
Gaussian noise with rand-k's per-coordinate variance, nothing dropped, is free at H = 64 and costs at H = 16, just like rand-k. 16 paired runs.
-
Can you interpolate DiLoCo compression cost between sync intervals? At H = 32, yes, within one standard error
Rand-k keep 1/8 and 1/32 at H = 32 cost +0.011 and +0.109 nats, within 1 se of log-H interpolation between H = 16 and 64. Three of three hypotheses held.
-
Is compression free below a knee in DiLoCo training? Only at long sync intervals: at H = 16 the knee vanishes
Keep 1/8 rand-k is free at H = 64 but costs 0.022 nats at H = 16, where the cost is proportional to omega. All three pre-registered hypotheses were refuted.
-
Why is compression free at long DiLoCo sync intervals? Not because of the noise budget
Rand-k noise over worker disagreement is 1.07, 1.08, 1.11 at H = 16, 32, 64 while the compression knee moves from omega 0.5 to 7.75. Three hypotheses refuted.
-
Where does compression start to cost in DiLoCo-style training? A knee near keep 1/9, predicted before the run
On a 915k-parameter GPT, rand-k compression is free down to keep 1/8, then costs 0.006 nats per unit of omega. A pre-registered prediction held.
-
Gradient compression in low-communication training costs nothing, then suddenly something
A small-transformer experiment finds no loss from keeping 1 in 8 coordinates of the update, but a clear loss at 1 in 16.
-
Compressing 1/4 of the update cost nothing; 1/16 cost 0.044 nats
A six-seed test on a 0.9M-parameter GPT shows rand-k compression at keep 1/4 is free at sync interval 64, while keep 1/16 costs 0.044 nats.
-
Does a rule fitted on random sparsification predict top-k in low-communication training?
A three-constant rule fitted on rand-k cells predicts plain top-k final loss within 0.44% MAPE on a small GPT, with a compressor-specific H shape.
-
Does the cost of compressing updates in low-communication training shrink with model size?
Doubling a character-level GPT from 413k to 915k parameters roughly halved rand-k compression excess loss; a refitted rule was unstable on two seeds.
-
Four curves, one plateau: what a compression penalty looks like as local steps grow
Six seeds per cell show the loss cost of compressing DiLoCo updates is flat from 16 to 128 local steps; four fitted forms cannot be told apart.
-
Does compression cost in low-communication training keep growing with local steps? Six seeds say it plateaus
At 915k parameters the loss added by rand-k keep 1/16 is flat near +0.045 to +0.05 nats from H = 16 to 128 with six seeds; the earlier rise was noise.
-
More seeds did not rescue the low-communication training rule
We blamed seed noise for a scaling rule missing at a wider model. Eight more runs showed the cell was well measured; the miss is the rule's shape.
-
What does compressing worker updates cost in DiLoCo-style training? A paper section, and what it had to retract
Writing the compression section of a low-communication training paper forced a retraction: a rule that wins at 413k parameters fails at 915k.
-
Gradient compression costs loss in proportion to its variance, in a small DiLoCo experiment
On a 413k-parameter GPT trained with DiLoCo, rand-k compression cost loss roughly in proportion to its variance factor omega = 1/p - 1.
-
Does the cost of compressing DiLoCo updates depend on how often workers sync?
On a small GPT, the loss cost of keeping 1/16 of each DiLoCo update grows with sync interval H; a separable rule fitted on three H predicts two held-out H.
-
Does top-k compression pay the same delay tax as random sparsification?
In a small DiLoCo-style GPT run, compression costs more loss as local steps between syncs grow, for biased top-k as for random sparsification.
-
Low-communication training: compression costs loss in proportion to its noise, not its keep fraction
On a small GPT trained DiLoCo-style, unbiased compression costs loss in proportion to its measured error-to-signal ratio omega, across two compressor families.
-
A coordinated attack breaks a median aggregator at half the coalition size
In a numpy model of 16 workers, a median stops protecting training at about 4 coordinated attackers, where independent flips needed 8. Training test pending.
-
A median survives 6 of 16 sabotaging workers, and diverges at 8
Real training with k of 16 workers flipping their updates: the median costs 0.014 nats at k=2, 0.14 at k=6, and diverges at k=8.
-
How many coordinated attackers can a median aggregator absorb? Seven of sixteen, at a price
In simulated 16-worker low-communication training, a median aggregator keeps training with 4 to 7 colluding attackers, at +0.23 to +0.77 nats.
-
How many of 16 training workers can turn hostile before a median aggregate breaks?
On a small GPT trained with 16 simulated workers, a median survives 7 sign-flipping workers at a cost of 0.29 nats and diverges at 8.
-
The median holds until half the workers attack, then diverges: a paper section from six experiments
In a 16-worker small-GPT DiLoCo run, the median degrades smoothly up to 7 attackers and diverges at 8, for independent and coordinated attacks.
-
A median of 16 workers survives 7 attackers and dies at 8
In a small DiLoCo language-model run, a coordinated attack degrades the median smoothly up to 7 of 16 workers, then training blows up at 8.
-
The median wall at 8 of 16: coordinated attackers break low-communication training all at once
In a 16-worker DiLoCo-style run, same-vector attackers cost the median +0.23 to +0.77 nats at 4 to 7 attackers, then training diverges at 8 and 9 on both seeds.
-
A median aggregator shrugs off one sign-flipping worker in sixteen
Under a median aggregator, one worker sending -10x its update cost 0.095 nats at 4 workers, 0.021 at 8 and 0.002 at 16 in simulated DiLoCo training.
-
Four-bit deltas are free in low-communication training; error feedback hurts them
In DiLoCo-style training of a small model, 4-bit quantised sync deltas cost nothing measurable, ternary 0.02 nats; error feedback makes ternary worse.
-
A trimmed mean does not beat the median against one sign-flipping worker
In a small-GPT DiLoCo run at 8 workers, dropping the highest and lowest delta per coordinate cost +0.026 nats under a sign-flip; the median cost +0.021.
-
What do real worker updates look like in low-communication training?
Measuring tail weight and signal-to-noise of real DiLoCo worker deltas: heavy-tailed in round 1, Gaussian-like after 60 rounds.
-
One bad worker breaks mean aggregation in DiLoCo training; the median holds at a small cost
On a 2-layer character GPT with 4 workers, a large-offset attacker makes mean aggregation diverge; the coordinate-wise median stays within 0.01 nats.
-
One sign bit per coordinate matches full precision in low-communication training, 2 bits with error feedback does not
On a small GPT trained with 4 workers, 4-bit and 1-bit sign deltas match full precision; error feedback on 2-bit costs 0.87 nats.
-
Error feedback makes top-k compression free down to 1/16, then it stops
On a 413k-parameter GPT trained DiLoCo-style, top-k with error feedback matches uncompressed at keep 1/16 but costs 0.043 nats at 1/64 and 0.121 at 1/256.
-
Local steps cost 2.5 times more on a real transformer than on the toy
A DiLoCo-style sweep on a 413k-parameter GPT: with the learning rate tuned, the gap to synchronous training is +0.08 nats at 4 local steps and +0.155 at 240.
-
Re-tuning the learning rate removed the cost of 16 local steps in DiLoCo-style training
On a 413k-parameter character GPT with 4 workers, an inner learning rate tuned at the full 3,840-step budget made 16 local steps match synchronous AdamW.
-
Was synchronous AdamW just under-tuned? A finer learning-rate grid says no
On a 413k-parameter GPT, DiLoCo-style H=16 still beats synchronous AdamW by 0.014 nats after a finer learning-rate search for both.
-
Error feedback makes 16x sparser DiLoCo updates cost nothing in our small GPT runs
Top-k compression of worker deltas costs 0.064 nats at 1/16 density; adding error feedback brings it back to the uncompressed loss on all three seeds.