Written by an AI agent working in this repository. CPU-scale study: character-level Shakespeare, a 915k-parameter GPT, M = 4 workers, 3,840 inner steps per worker.
The last step ran a compression recipe for DiLoCo at a sync interval of H = 32. The recipe compresses every sync with rand-k except the last one, which goes dense. At keep 1/4 it ended 0.006 nats below uncompressed training, with a standard error of 0.008, so four seeds could not say whether the gain was real. The seeds also disagreed more at H = 32 than at H = 16 or 64. This step added four more seeds, 4 to 7, for uncompressed training and for keep 1/4 and keep 1/8.
The test
The plan and six hypotheses were committed before any run. The training code is the previous step's, byte for byte. Only the seeds changed. Two of the new uncompressed runs had been stored by an earlier milestone, and both reran bit-exactly.
What happened
Final loss minus uncompressed training on 64 fresh validation batches, paired by seed. Each cell is the mean, with the standard error in brackets:
| keep (omega) | seeds 0-3 | seeds 4-7 | all 8 |
|---|---|---|---|
| 1/4 (3) | −0.006 (0.008) | −0.009 (0.003) | −0.007 (0.004) |
| 1/8 (7) | +0.002 (0.009) | +0.004 (0.002) | +0.003 (0.004) |
- Keep 1/4: probably a small gain, still not resolved. All four new seeds came in below uncompressed, so 7 of 8 are below zero. The pooled mean is 1.8 standard errors from zero. That misses the 2 se bar set in advance, so the registered test is refuted. The exact sign-flip permutation P is 0.13.
- Keep 1/8 ties. It sits 0.6 se from zero, as predicted.
- The wide spread came from two seeds. On the new seeds the spread is a third to a fifth of what it was on seeds 0 to 3. The one positive keep-1/4 seed is seed 1, where uncompressed training is unusually good: its fresh loss is 1.593, against 1.609 to 1.633 on the other seven. Both compressed arms look bad on that seed (correlation r = 0.53 across seeds), which fits a lucky reference run more than a bad compressed one.
- The cost of skipping the dense last sync is 0.0011 nats per unit of omega on the new seeds and 0.0013 pooled. That is slightly below the 0.0012 to 0.0020 band predicted, so it is not quite at its H = 64 value yet.
What it means
If you compress DiLoCo's syncs with rand-k keep 1/4 and send the last sync dense, the final model was never measurably worse than uncompressed training at any H tried: 16, 32 or 64. At H = 32 it lands about 0.007 nats below, which eight seeds cannot yet separate from zero. It sends a quarter of the bytes on every sync but the last. Keep 1/8 ties at H = 32 and needs H = 64 to gain.
The next step goes after the reference run, not more compressed seeds: a second uncompressed run per seed, so each comparison is against the average of two baselines.
Limits: one model width, eight seeds, and fresh batches drawn from the validation text rather than a held-out split.
Code, data and the pre-registered plan: research/lowcomm-training, LOG entry "Milestone 6ad", script experiments/gpt_scale/recipe_6ad.py.