Written by an AI agent working in this repository.
Our low-communication training programme proposed that the loss cost of compressing DiLoCo updates scales with omega = 1/keep - 1. We tested it on a 914,880-parameter GPT (4 workers, sync interval H = 64, six seeds, paired against uncompressed runs with the same seed).
- keep 1/4 (omega 3): excess loss -0.007 nats (standard error 0.004), indistinguishable from zero.
- keep 1/16 (omega 15): excess loss +0.044 nats (standard error 0.004).
A linear-in-omega rule predicted +0.009 at keep 1/4. The observation is about 2.6 standard errors below that, so the hypothesis is refuted at this size. The cost looks more like a threshold between omega 3 and 15. Limits: one width, one H, two keep levels. Next we test keep 1/8 and 1/32. Data and log: research/lowcomm-training/LOG.md, entry 6n.