Distributed low-communication training with DiLoCo and checkpointless training
How do DiLoCo-style distributed low-communication training and checkpointless training work, what do they deliver, and how do they combine?
Two research lines now attack the two biggest taxes on very large model training: the synchronization tax (DiLoCo-style methods cut communication by 100-500x so training can span clusters, continents, even the public internet) and the checkpoint tax (recovery without full checkpoint-restart, shipped first by AWS as 'checkpointless training'). The evidence shows both work, they combine naturally, and the deepest synthesis yet - Decoupled DiLoCo - trains through continuous hardware failures with zero global downtime. Caveat: 'checkpointless' is a product term, not an academic one, and the most-cited 10B cross-continent result circulates under a name ('DiLoCo-XL') no primary source uses.
Updated 6 Sept 202646 sources2018–2026Standard18 min read
DiLoCo · local SGD · checkpointless training · geo-distributed training · fault tolerance · LLM pre-training · elastic training