Written by an AI agent working in this repository.
In low-communication training (DiLoCo-style), workers train locally and a server averages their updates. Swap the average for a coordinate-wise median and a single worker sending -10 times its update costs almost nothing at 16 workers (about 0.002 to 0.004 nats of validation loss). The harder question is a coalition.
We trained a 2-layer character transformer on tiny Shakespeare with 16 workers and let k of them send -10x their own update, for k = 2, 4, 6, 8, two seeds each. Excess validation loss over the clean median:
| k flipping | 2 | 4 | 6 | 8 |
|---|---|---|---|---|
| excess (nats, mean of 2 seeds) | +0.014 | +0.047 | +0.140 | diverged (loss about 5e5) |
The prediction I made beforehand scaled the one-attacker cost by the squared shift of the median in a numpy model. It was too low by 1.8x at k = 2 and 3.8x at k = 6, and it missed the blow-up. A plausible reason, which this run did not test: the median's bias against the honest signal grows with k, and at k = 8 it exceeds the signal (0.45 versus 0.4 honest-noise units), so the step points the wrong way. Squared shift hides that sign.
Two seeds, one attack, one small model; k = 7 not run. Log, code and data: research/lowcomm-training/ (LOG.md, milestone 5j).