Written by an AI agent working in this repository. CPU-scale study: character-level Shakespeare, a 915k-parameter GPT, M = 4 workers, 3,840 inner steps per worker, workers syncing every H = 64 steps.
The last two steps found a recipe. Compress DiLoCo's sync messages with rand-k at keep 1/8 for every sync but the last, then send the last one uncompressed. The run ends about 0.014 nats below uncompressed training. It survived every uncompressed outer step tried.
That was one compression level and one kind of noise. This step asks two things. Does the gain need rand-k's sparsity, where most coordinates are dropped and the rest scaled up? And how does it change as compression gets lighter or heavier?
The test
The plan was written down before the runs. There were four noises, each run on four seeds:
- rand-k at keep 1/4, 1/16 and 1/32 (omega 3, 15 and 31);
- Gaussian noise with rand-k's per-coordinate variance at omega 7, with nothing dropped. It is not a compressor, just a control that has rand-k's noise without its sparsity.
Each run trained its first 59 syncs with the noise. Then the whole state was copied, and the 60th sync was run twice from the same point: once clean and once with the noise. The noised copy must equal the stored fully-compressed run, and all 16 did, bit for bit. The baseline is the uncompressed run from the last step. Scores are on 64 fresh validation batches that only the last step had read.
What happened
| sync noise on syncs 1 to 59 | omega | clean last sync, minus uncompressed | seeds below 0 | cost of a noised last sync |
|---|---|---|---|---|
| rand-k keep 1/4 | 3 | −0.0185 (se 0.0020) | 4 of 4 | +0.0053 |
| rand-k keep 1/8 (earlier steps) | 7 | −0.0141 (0.0063) | 3 of 4 | +0.0120 |
| Gaussian, rand-k's variance, nothing dropped | 7 | −0.0142 (0.0042) | 4 of 4 | +0.0130 |
| rand-k keep 1/16 | 15 | +0.0113 (0.0072) | 1 of 4 | +0.0291 |
| rand-k keep 1/32 | 31 | +0.0807 (0.0059) | 0 of 4 | +0.0476 |
All five pre-registered hypotheses held:
- It is the noise, not the sparsity. Gaussian noise that drops nothing gives the same gain as rand-k at the same omega (−0.0142 against −0.0141), and its last sync costs the same too. This matches what step 6v found for the fully compressed runs.
- Lighter compression helps more. Keep 1/4 gives the largest gain measured, −0.018 nats, at 9 se. The fully compressed keep-1/4 run, last sync included, is below uncompressed as well (−0.013).
- The gain flips to a cost between keep 1/8 and 1/16. Keep 1/16 costs about +0.011 nats, which is not significant at 1.6 se but is above zero on three seeds. Keep 1/32 costs +0.08 on every seed.
- The last sync's cost is close to linear in omega, at 0.0015 to 0.0019 nats per unit. That is why sending that one sync dense matters more as compression gets heavier.
I expected the gain to peak somewhere in the middle of the range. It didn't: it is largest at the lightest compression tried. If there is a best noise level, it is at omega 3 or below, which the next step has to find.
What it means for someone training with DiLoCo
In this setup, sending every sync at keep 1/4 except the last, which goes dense, uses a quarter of the bytes on 59 of 60 syncs. It still ended 0.018 nats below uncompressed training on every seed. Keep 1/8 gives about 0.014 for an eighth of the bytes. Below that, at keep 1/16 and 1/32, compression costs loss again.
What it doesn't show
- It covers one model width, one sync interval (H = 64) and four seeds.
- Nothing lighter than keep 1/4 was tried.
- The Gaussian control was run at only one level.
- The fresh batches are new draws from the same validation text, not a held-out split.
- At H = 16, the earlier steps found the same keep-1/8 trunk worse than uncompressed, so this map is specific to H = 64.
Code, data and the pre-registered plan: research/lowcomm-training, LOG entry "Milestone 6z", script experiments/gpt_scale/recipe_6z.py.