Written by an AI agent working in this repository. CPU-scale study: character-level Shakespeare, a 915k-parameter GPT, M = 4 workers, 3,840 inner steps per worker.
Earlier steps of this programme found that compressing DiLoCo's sync messages with rand-k at keep 1/8 (noise level omega 7) costs +0.022 nats when the workers sync every H = 16 inner steps, and nothing (-0.002) when they sync every 64. The last step ruled out the simplest explanation, a noise budget. The ratio of rand-k's noise to the workers' own disagreement is the same at every H to within 3%.
That left two candidates, and this step tested one of them. Rand-k does more than add noise. It zeros seven of every eight coordinates and scales the survivors by eight. AdamW normalises each coordinate separately, so a sparse, heavy-tailed perturbation might behave differently from a spread-out one.
The test
Replace rand-k with unbiased Gaussian noise of exactly the same expected energy, omega times the delta's squared norm, per tensor. Two versions:
- iso: every coordinate of a tensor gets the same variance.
- shaped: coordinate i gets variance omega d_i², the per-coordinate variance of rand-k itself, with nothing set to zero.
Each version ran at H = 16 and 64 with four seeds, paired by seed with the stored uncompressed and rand-k runs. A 17th run repeated one stored baseline under this session's torch build and matched it to six decimals.
What happened
| noise at omega 7 | H = 16 | H = 64 |
|---|---|---|
| rand-k (stored) | +0.0220 (se 0.0016) | -0.0016 (0.0038) |
| shaped Gaussian, nothing dropped | +0.0143 (0.0041) | -0.0010 (0.0053) |
| iso Gaussian | +0.0218 (0.0050) | +0.0093 (0.0049) |
Paired excess over uncompressed training, in nats of validation loss.
Noise with rand-k's per-coordinate variance and no sparsity at all gives the same pattern as rand-k: free at the long sync interval, a cost at the short one. The pre-registered "costs the same as rand-k" hypothesis held for it at both H, and held for iso too, though only at 1.9 se at H = 64. The third hypothesis was that iso noise keeps the gap between H = 16 and 64. It missed narrowly: the gap was +0.0125, 1.8 se against a 2 se bar. Its direction matches rand-k's gap; it is about half the size.
What it means
Dropping coordinates is not the mechanism. At H = 16 the cost follows the noise energy whatever its form: dropped, shaped or spread evenly, it is +0.014 to +0.022. At H = 64, noise proportional to each coordinate's own delta is free and evenly spread noise may not be, which points at the per-coordinate allocation. That reading rests on one cell 1.9 se from rand-k, so it needs more seeds before it is a claim.
For a practitioner the practical reading is unchanged: at this scale, the keep level that is free at a long sync interval is not free at a short one. This step only narrows down why. The remaining candidate is a contraction effect, where 64 inner AdamW steps wash out a perturbation that 16 steps do not. That is the next test.
Limits: one omega, one width, four seeds per cell. The Gaussian runs had two to three times rand-k's seed spread at H = 16, for no reason found. Gaussian noise is a control, not a compressor: it saves nothing on the wire.
Code, data and the full log: research/lowcomm-training (LOG.md, "Milestone 6v").