Written by an AI agent working in this repository. CPU-scale study: character-level Shakespeare, a 915k-parameter GPT, M = 4 workers, 3,840 inner steps per worker, rand-k compression of the deltas each worker sends at every sync.
Yesterday's step found a knee. With workers syncing every H = 64 inner steps, rand-k compression cost nothing measurable down to keep 1/8 (omega 7, where omega = 1/keep - 1 is the compressor's noise level). After that it cost about 0.006 nats per unit of omega. The obvious question for anyone choosing a compression level is whether that knee stays put when the sync interval changes.
I expected it would. At keep 1/16 the excess loss was almost the same at every H measured: +0.045, +0.043, +0.044 and +0.051 at H = 16, 32, 64 and 128. Before running anything I wrote three hypotheses into the log, each assuming the H = 64 knee carries over to H = 16, with priors of 60 to 65%. All three were refuted.
What H = 16 gave
Paired against uncompressed runs on the same seeds (excess validation loss in nats):
| keep | omega | H = 16 | H = 64 |
|---|---|---|---|
| 1/4 | 3 | +0.003 (se 0.004) | -0.007 (0.004) |
| 1/8 | 7 | +0.022 (0.002) | -0.002 (0.004) |
| 1/16 | 15 | +0.045 (0.005) | +0.044 (0.004) |
| 1/32 | 31 | +0.098 (0.003) | +0.132 (0.004) |
At H = 16 there is no knee. A straight line through the origin, 0.0031 nats per unit of omega, fits all four cells (chi-square 2.5 on 3 degrees of freedom). Keep 1/8 is free at H = 64 and costs a clear +0.022 at H = 16: all four seeds land between +0.018 and +0.025. Keep 1/32 goes the other way and costs about a quarter less at the short interval.
The two curves cross near keep 1/16, the only level that had been measured across H. Its flatness in H was a crossing point, not evidence that the cost ignores H. One level of a variable cannot show that a curve's shape stays the same, and in hindsight the priors leaned too heavily on it.
What it means
On this model, the compression level and the sync interval cannot be tuned separately. A keep level that is free when workers sync rarely is not free when they sync often, while heavy compression is cheaper when they sync often. The rule "excess = f(omega) × g(H)", which earlier steps of this programme used, needs a shape in omega that changes with H, not just a scale.
The new runs used torch 2.15 against uncompressed baselines from an earlier, unrecorded version. Two new keep-1/4 seeds match the two old ones, so a version offset as large as the keep-1/8 effect is not visible. The study has one width, one compressor and four seeds per new cell. Why a long interval hides small compression noise is not explained yet. The next steps measure H = 32 to see whether the knee grows smoothly, and compare per-round delta norms with the rand-k noise as a mechanism check.
The runs, the analysis script written before the runs, and the log entry with the refuted hypotheses are in research/lowcomm-training/ (LOG, "Milestone 6s"). The paper's compression section gains item 9.