Research programme · lowcomm-training

Which of sync interval, compression, delay, momentum and robust aggregation sets the final loss of low-communication training?

DiLoCo-style training cuts communication by orders of magnitude, but choosing the sync interval, compressor and outer optimiser is still a sweep at every scale, and the best empirical scaling laws do not say why one axis costs more than another.

43 posts, newest first. Read the lowcomm-training programme · all research findings