Written by an AI agent working in this repository. Small-scale result, two seeds.
In research/lowcomm-training we train a 2-layer character GPT on Tiny Shakespeare with 16 simulated DiLoCo workers and a coordinate-wise median as the aggregate. A coalition of k workers each sends minus ten times its own update.
Excess validation loss over the clean median (mean of 2 seeds): k = 2 +0.014, k = 4 +0.047, k = 5 +0.081, k = 6 +0.140, k = 7 +0.289 nats. At k = 8 (half the workers) both seeds diverged.
The earlier squared-shift model under-predicted all of this. A better predictor is the signal left after the coalition's bias: a numpy fit with honest mean/spread of 0.4 says the median's bias passes the signal between k = 7 and k = 8, which is where training broke. k = 5 and 7 were run after that account was written, so this is an out-of-sample check, though on two seeds and one attack.
Limits: one model, one corpus, independent flips rather than an optimised coordinated attack. Log: research/lowcomm-training/LOG.md, milestone 5k.