Written by an AI agent working in this repository. CPU-scale study: character-level Shakespeare, a 915k-parameter GPT, M = 4 workers, 3,840 inner steps per worker, workers syncing every H = 64 steps.
The last step found a recipe that trains better than sending every sync uncompressed. Compress DiLoCo's sync messages with rand-k for every sync but the last, then send the last one dense. At keep 1/4 the run ended 0.018 nats below uncompressed training. At keep 1/8 it ended 0.014 below, at keep 1/16 it was 0.011 worse, and at keep 1/32 it was 0.081 worse. The best setting was the gentlest one tried. The gain has to vanish as the noise goes to zero, though, so somewhere below keep 1/4 it must turn back. Where?
The test
The plan was written down before any run. The recipe ran at rand-k keep 1/2 (omega 1), four seeds. It also ran with Gaussian noise of rand-k's per-coordinate variance and nothing dropped, at omega 3 and 1 with four seeds each, and at omega 1/3 (keep 3/4's noise level) with three seeds as an exploratory arm. Every run was scored on 64 validation batches that only the last two steps had read. A control rerun of one earlier run matched it bit-exactly.
What happened
Final loss minus uncompressed training, fresh batches, seed mean (standard error):
| noise level (omega) | rand-k | Gaussian, same variance |
|---|---|---|
| 1/3 (keep 3/4's noise) | −0.001 (0.006), 3 seeds | |
| 1 (keep 1/2) | −0.012 (0.006) | −0.007 (0.002) |
| 3 (keep 1/4) | −0.018 (0.002) | −0.018 (0.003) |
| 7 (keep 1/8) | −0.014 (0.006) | −0.014 (0.004) |
| 15 (keep 1/16) | +0.011 (0.007) |
The gain rises from nothing at omega 1/3 to its largest at omega 3, then falls, and it is gone by omega 15. Three findings:
- The sweet spot is around keep 1/4. The two pre-registered tests at keep 1/2 both fell just short of their bars. The gain over uncompressed training came in at 1.98 standard errors against a bar of 2. Keep 1/2 minus keep 1/4 came in at 1.2. Rand-k and the Gaussian noise turned out interchangeable, so pooling them per seed is fair, though I did that after the fact. Pooled, keep 1/4 beats keep 1/2 by 0.009 nats at 3.9 standard errors, on all four seeds.
- It's the noise, not the sparsity, at a second compression level. At omega 3, Gaussian noise that drops nothing matches rand-k on every measure: the gain, the last sync's cost and even the weight norm, to five figures. That held at omega 7 too.
- The last sync's own cost is a straight line. The cost of sending the final sync compressed instead of dense is 0.0013 to 0.0019 nats per unit of omega across nine settings, from omega 1/3 to 31.
What it means
For a model and sync interval like these, "compress gently to be safe" is the wrong instinct. Keep 1/2 sends twice the bytes of keep 1/4 and gets about half the gain. Even with every sync compressed, keep 1/4 still ends 0.013 below uncompressed.
What it doesn't show
- It is one model width at one sync interval (H = 64). At H = 16 compression costs instead of helps, and it is not known yet whether a sweet spot survives there.
- Keep 1/2 against keep 1/4 is settled only by the after-the-fact pooled test.
- There are four seeds, and three at the smallest noise.
- The fresh batches are new draws from the validation text, not a held-out split.
Next: the same map at H = 32 and 16, then a held-out split.
Code, data and the pre-registered plan: research/lowcomm-training, LOG entry "Milestone 6aa", scripts experiments/gpt_scale/recipe_6aa.py and pooled_6aa.py.