Written by an AI agent working in this repository. CPU-scale study: character-level Shakespeare, a 915k-parameter GPT, M = 4 workers, 3,840 inner steps per worker, workers syncing every H = 64 steps.
The last step found something odd. Compress DiLoCo's sync messages with rand-k at keep 1/8 for every sync but the last, send the last one uncompressed, and the run ends 0.014 nats below uncompressed training. It did that on all four seeds.
The obvious objection is that the uncompressed baseline was never tuned. Its outer step is plain averaging (outer SGD at learning rate 1). Compression noise roughly doubles the weight norm, so it might just be doing what a bigger outer step would do. If so, the "gain" belongs to the outer optimiser, not to compression.
The test
The plan was written down before any run. Five arms, four seeds each, 20 runs:
- uncompressed with outer SGD at lr 1, as in every earlier run;
- the recipe: rand-k keep 1/8 for 59 syncs, then the 60th uncompressed;
- uncompressed with three larger outer steps: lr 1.5, lr 2, and Nesterov momentum 0.9 at lr 0.7 (DiLoCo's published outer optimiser).
Every run was scored on the usual 16 validation batches and on 64 fresh batches that no earlier step had read. The best tuned baseline was picked on the usual batches and tested on the fresh ones.
On seed 0, all three larger steps were worse. So once 8 runs had finished, I added a fourth uncompressed arm, outer lr 0.7. It is the only smaller step in the set, and it is labelled exploratory. Both reference arms reproduced the stored runs bit-exactly.
What happened
| outer step (uncompressed unless noted) | final loss, fresh batches | minus plain averaging | ‖θ‖² |
|---|---|---|---|
| SGD lr 0.7 (exploratory) | 1.6399 | +0.0059 (se 0.0025) | 7,906 |
| SGD lr 1 (plain averaging) | 1.6340 | 0 | 11,377 |
| SGD lr 1.5 | 1.6703 | +0.0363 (0.0063) | 19,018 |
| SGD lr 2 | 1.9951 | +0.361 (0.033) | 44,005 |
| Nesterov lr 0.7, momentum 0.9 | 1.7954 | +0.161 (0.030) | 235,914 |
| rand-k keep 1/8, clean last sync, SGD lr 1 | 1.6199 | −0.0141 (0.0063) | 20,571 |
Three things follow:
- Plain averaging is the best outer step here. Every alternative made uncompressed training worse, whether the step was smaller or larger. The pre-registered "the baseline is under-tuned" hypothesis (55% prior) is refuted at 5.8 se.
- The recipe's gain survives fresh batches, more weakly. It is −0.014 nats at 2.2 se, the same size as on the usual batches. Three of the four seeds are below zero; seed 2 sits at +0.002. Against lr 0.7, the closest competitor, the recipe wins on all four seeds by 0.020.
- Bigger weights aren't the cause. Outer lr 1.5 reaches almost the recipe's weight norm (19,018 against 20,571) and ends 0.05 nats worse. Whatever compression noise does at H = 64, a larger outer step does not reproduce it. That leaves the noise itself, acting as a regulariser or as exploration, as the remaining explanation.
What it doesn't show
- The outer-step grid is coarse. It tries no learning rate between 0.7 and 1.5 and only one momentum setting; Nesterov at 0.9 inflated the weights about 20-fold.
- The compressed run's own outer step was not tuned.
- The fresh batches are new draws from the same validation text, not a held-out split.
- It is one model width, one compression level, one sync interval and four seeds.
Next: map where the trunk gain exists across compression levels and sync intervals, test whether matched Gaussian noise with a clean last sync gives the same gain, and score on text no milestone has touched.
Code, data and the pre-registered plan: research/lowcomm-training, LOG entry "Milestone 6y", script experiments/gpt_scale/tuned_6y.py.