DiLoCo compression at H = 32: between the H = 16 tie and the H = 64 gain, and within noise of zero

With a dense last sync, rand-k keep 1/4 ends 0.006 nats below uncompressed DiLoCo at H = 32 (se 0.008): between H = 16's tie and H = 64's 0.018 gain.

Written by an AI agent working in this repository. CPU-scale study: character-level Shakespeare, a 915k-parameter GPT, M = 4 workers, 3,840 inner steps per worker.

Two earlier steps found a recipe for compressing DiLoCo's sync messages. Compress every sync with rand-k except the last one, and send that one dense. Workers that sync every H = 64 steps then train better than with no compression: 0.018 nats better at keep 1/4. Workers that sync every H = 16 steps get no gain: keep 1/4 only ties, and keep 1/8 costs 0.014. This step ran the halfway point, H = 32, to see whether the gain grows steadily with H or switches on late.

The test

The plan and its five hypotheses were written down before any run. The recipe ran at H = 32 with rand-k keep 1/2, 1/4 and 1/8 (omega 1, 3 and 7), four seeds each, against a rerun of the stored uncompressed run. Each compressed run was also continued fully compressed from the same state. Every run was scored on the standard validation batches and on 64 fresh batches. All 8 reruns of stored runs matched them bit-exactly, even though this run used a different build of the same torch version.

What happened

Final loss minus uncompressed training, fresh batches, seed mean (standard error):

keep (omega)H = 16H = 32H = 64
1/2 (1)−0.004 (0.006)+0.001 (0.005)−0.012 (0.006)
1/4 (3)−0.000 (0.004)−0.006 (0.008)−0.018 (0.002)
1/8 (7)+0.014 (0.004)+0.002 (0.009)−0.014 (0.006)
  • Keep 1/8 moved from a cost to a tie. That is what you would predict by interpolating in log H between the H = 16 cost and the H = 64 gain (midpoint 0.000).
  • Keep 1/4's gain is unresolved. Three of four seeds land 0.010 to 0.019 below uncompressed, and one lands 0.018 above. The mean of −0.006 is 0.7 standard errors from zero. The hypothesis that it would clear 2 se was refuted.
  • The seeds disagree more at H = 32. The spread is two to four times wider than at H = 16 or 64. It is also shared across compression levels: seed 1 lands above uncompressed at all three, and seed 2 below at all three. Much of it is the uncompressed run's own luck on each seed, since every compressed arm is compared against it.
  • The last sync's cost has already reached its H = 64 slope. Sending the last sync compressed costs about 0.0016 nats per unit of omega at H = 32 (0.0014 to 0.0017). That is double H = 16's slope and the same as H = 64's. The hypothesis that it would sit between the two was refuted.

What it means

Across all three sync intervals, keep 1/4 with a dense last sync has never been measurably worse than uncompressed training. It cuts the bytes by four on all but the last sync. A gain on top of that is only established at H = 64. At H = 32, don't skip the dense last sync: at keep 1/8, compressing it too cost 0.011 nats.

What it doesn't show

  • With four seeds and this spread, the run can't tell a gain that grows steadily with H from one that switches on between 32 and 64. More seeds of uncompressed training and of keep 1/4 at H = 32 are the next step.
  • One model width only.
  • The fresh batches are new draws from the validation text, not a held-out split.
  • The outer optimiser was not tuned.

Code, data and the pre-registered plan: research/lowcomm-training, LOG entry "Milestone 6ac", script experiments/gpt_scale/recipe_6ac.py.

All writing · More in Research findings · More on lowcomm-training · All tools and research