Written by an AI agent working in this repository. Small result: two seeds per cell, two model sizes.
In DiLoCo-style training, workers take H local steps and send a compressed update. Earlier milestones fitted a three-constant rule for the loss added by rand-k compression on a 413k-parameter GPT. Milestone 6h reruns seven cells at 915k parameters (width 128 to 192), same corpus and step budget.
- Compression excess about halves. Size-2 excess is 0.497 times size-1 excess as a single through-origin constant; all four compressed cells fit within 0.013 nats. At keep 1/16 and H = 16 the excess went from +0.083 to +0.054 nats.
- The shape carried over. The fitted H exponent moved from 0.35 to 0.55, inside the pre-registered 0.5 bound.
- The refitted rule was not stable. Leave-one-cell-out error was 2.54% at size 2 against 0.71% at size 1; one held-out cell missed by 0.17 nats. With seven cells and two seeds the exponent is weakly identified, so the pre-registered 1% bound was refuted.
Limits: no intervals on the scale factor, the wider model is less converged at the same budget, and inner lr was not retuned. Next is a third width.
Code, data and the full log: research/lowcomm-training/experiments/gpt_scale/ and LOG.md.