Written by an AI agent working in this repository. CPU-scale study: character-level Shakespeare, a 915k-parameter GPT, M = 4 workers, 3,840 inner steps per worker.
This programme has a puzzle it keeps narrowing down. Compressing DiLoCo's sync messages with rand-k at keep 1/8 (noise level omega 7) costs +0.022 nats when workers sync every H = 16 inner steps. It costs nothing measurable (-0.002) when they sync every 64. Two explanations have already failed. The noise budget is the same at every H. Rand-k's sparsity is not it either, because Gaussian noise with nothing dropped behaves the same way.
The last candidate was contraction. Perhaps 64 AdamW steps after a noisy outer step wash the noise out, and 16 steps do not.
The test: fork the run and inject one round of noise
Train uncompressed to the middle of the budget (step 1,920). Copy everything: the global model, the four workers, their AdamW moments and their data streams. Run the next round in both copies. Both copies take the same inner steps. One averages the deltas as they are, and the other averages rand-k compressed deltas. The two global models now differ by exactly one round of real compression noise. Then both copies continue uncompressed on identical data for 256 inner steps, and we measure how far apart they are after every round.
A control fork with no compression in either copy stayed identical to the last bit. That confirms the copy is complete.
What happened
| H = 16 | H = 64 | |
|---|---|---|
| injected energy per inner step | 1.96 | 2.07 |
| share of it left after 64 inner steps | 100.6% | 100.0% |
| share left after 256 inner steps | 107% | 106% |
| loss cost right after the injection | +0.0063 nats | +0.0121 nats |
| loss cost 256 inner steps later | -0.0000 | +0.0023 |
Seed means of 4 runs each. Standard errors are about 0.001 nats on the loss rows.
Nothing washes out. In parameter space the injected noise is carried along at both sync intervals and even grows a little. One registered hypothesis technically held: after 64 inner steps less noise survives at H = 64. But the margin is 0.6% of the energy, which cannot turn +0.022 nats into zero. The other two hypotheses were refuted.
The loss tells a different story from the parameters. The loss cost does fade, so most of the noise sits in directions the loss barely cares about. It fades more slowly at H = 64, and it starts twice as large there.
What it means
Per round and per step, H = 64 looks worse. It takes the same noise rate, pays more per injection and forgets more slowly. Yet a run compressed at that level throughout is free at H = 64. The costs of single injections therefore do not add up to the cost of compressed training. A compressed H = 64 run ends right after a fresh round of rand-k noise. If that round cost what it costs at mid-training, the run could not end at -0.002.
So the knee is a whole-trajectory effect. Either a single round costs much less at the end of training than in the middle, or a run that is compressed throughout adapts to the noise, perhaps through the inner optimiser's moments. A one-off injection cannot show that adaptation. The next test injects at the last round. A reverse test then removes compression from the last round of a compressed run.
Limits: one injection point, one noise level, one model width and four seeds per H. The loss was read only at the injection and at the window's end.
Code, data and the full log: research/lowcomm-training (LOG.md, "Milestone 6w").