Distributed low-communication training with DiLoCo and checkpointless training

Why this question

Two costs dominate the economics of very large model training. The first is the synchronization tax: data-parallel training all-reduces gradients every step, which is why accelerators must sit behind NVLink/InfiniBand in one building. That forces single-megacampus builds, long power-procurement cycles, and stranded capacity everywhere else. The second is the checkpoint tax: a 10,000-GPU run fails constantly — failures and stragglers are “the norm rather than the exception” in production experience ³³ — and each recovery means writing and reloading terabytes of state, stalling every GPU for minutes to hours, and paying an energy bill for the privilege ⁴³.

For anyone reasoning about where AI training can physically happen — one giant site versus many smaller ones, firm power versus renewable curtailment windows, owned clusters versus rented and preemptible ones — these two taxes are the whole game. If low-communication training removes the need for co-location, and checkpointless recovery removes the need for persistent-state safety nets, then training becomes schedulable across time and geography the way inference already is. This review asks what each line delivers, where the evidence stops, and whether the combination is real. It sits alongside this library’s reviews of interconnect economics, large-batch convergence, tokens-per-watt and model-to-grid datacenter design ([/reviews/llm-interconnects/], [/reviews/batch-size-convergence-llm/], [/reviews/tokens-per-watt-benchmarks/], [/reviews/model-to-grid-ai-datacenter-efficiency/], [/reviews/mlperf-training-closed-optimization/]).

Scope and methods

The question was split into three research waves run in parallel: (1) the DiLoCo academic lineage (original method, scaling behaviour, variants, local-SGD theory roots), (2) the checkpoint-elimination literature (literal “checkpointless” work, elastic/recovery systems, checkpoint-cost quantification), and (3) the 2025–2026 frontier (multi-datacenter training, production adoption, energy/geography economics, combinations of the two lines). Searches ran over the arXiv API, OpenAlex, Semantic Scholar citation graphs, and Google News RSS (literal strings in the frontmatter audit strip); industry material came from primary vendor pages fetched in-session. Screening: 321 unique arXiv records were pooled across the waves, 52 candidates were abs-page-verified (title, authors, date, abstract fetched directly), and 46 were included; the rest were dropped as redundant variants or peripheral. Every included record has a verified DOI (10.48550/arxiv.*, all resolve). Five load-bearing papers were read in full text; the rest are abstract-level, and claims are hedged accordingly. One term was searched exhaustively and found not to exist: no arXiv paper, OpenAlex record, news item, or official post titled “DiLoCo-XL” was locatable on 2026-09-06 — treated as a finding, not an error (see Gaps). The “checkpointless training” label likewise has zero academic footprint; the substance lives under other names and in one AWS product, covered via the two-layer model: citable papers for mechanisms, inline-linked vendor documents for product claims (all attributed vendor-stated).

The landscape

The literature splits cleanly into two bodies that barely cite each other until 2026. The DiLoCo line grows out of decades of local-SGD and federated-averaging theory ¹⁶¹⁷¹⁸, became a method in late 2023 ¹, and exploded in 2025–2026 into a dense family: scaling studies ², communication-overlap tricks ³⁴, optimizer surgery ⁵¹⁴⁸, desynchronized and gossip variants ⁹¹³²², and — most importantly — an asynchronous, failure-tolerant redesign ⁷. A parallel community-driven line has actually shipped multi-continent and internet-scale runs ²³²⁴²⁶, recently surveyed as a field of its own ⁴⁶.

The checkpoint body is older (HPC checkpoint-restart), but the LLM era reframed it: production systems reports quantify the failure regime ³³, and a dense 2024–2026 crop of systems papers attacks checkpoint cost from every angle — lazy async ³⁵, unified/reshardable formats ³⁶, compression ⁴⁴, erasure-coded replication ³⁴, in-memory and hot-swap recovery ³⁸³⁹, checkpoint-free state migration ⁴⁰, and preemptible-instance training ⁴¹. The two bodies finally meet in 2026: Meta’s torchft library supports DiLoCo-style training with in-memory state redundancy, AWS sells “checkpointless training” for ordinary synchronous workloads, and DeepMind’s Decoupled DiLoCo is explicitly framed as absorbing hardware failure without global stalls. That convergence is the story this review is organized around.

Theme 1 — The theory roots: local SGD was always about the communication tax

The mathematical core of DiLoCo is decades-old: run SGD locally on each worker and average occasionally. The first clean theory showed local SGD converges at the same rate as mini-batch SGD per gradient evaluated — the communication saving is almost free in the convex regime ¹⁶. Empirically, “post-local” SGD was shown to fix the generalization collapse of very large mini-batches at equal time-to-accuracy ¹⁷.

But the theory also drew the boundary lines that still shape the field. Local SGD strictly dominates minibatch SGD only in narrow (quadratic) regimes; for general convex objectives it can be strictly worse, and data heterogeneity is the killer ¹⁸¹⁹. A production-scale empirical study found local SGD’s accuracy loss grows with scale — the opposite of what small experiments suggested ²⁰. Two consequences carried into the LLM era: naive periodic averaging is fragile, and the outer loop (how often you average and with what optimizer) matters as much as the local training. Later work on local SGD for LLMs found competitive results at equal compute and began mapping when multi-cluster training works ²¹, and a 2025 theory paper finally analysed the outer optimizer itself — learning rate, momentum, acceleration — as the lever that the earlier literature left unexamined ¹⁵.

Theme 2 — DiLoCo: the method that changed the operating point

DiLoCo reframed local SGD for the “islands of accelerators” setting: a small number of workers (initially 8), each itself a large cluster, synchronized rarely ¹. The design is a variant of federated averaging with two crucial choices: the inner optimizer on each worker is AdamW, and the outer optimizer that consumes the averaged parameter differences (“pseudo-gradients”) is Nesterov momentum — not plain averaging. Workers exchange state once every 500 steps, so where standard data-parallel training communicates every few hundred milliseconds, DiLoCo communicates every few minutes. On C4 with 8 workers the paper reports matching fully synchronous quality while communicating 500× less, plus robustness to the data distribution of each worker ¹.

The original authors explicitly predicted DiLoCo would look better at larger scale, because there is less interference during outer-gradient averaging. That prediction, unusually, was tested rather than asserted.

Theme 3 — Scaling behaviour: 4B, 10B, and the many-worker degradation problem

A 2025 scaling-law study did the careful thing: fixed compute budget, models from 35M to 10B parameters, and a search over replicas, hyperparameters and token budgets. It found DiLoCo scales predictably and robustly with model size; when well-tuned it scales better than data-parallel with size, tolerates larger batches, and can even win at small sizes. The headline validations were models of 4B and 10B parameters, where DiLoCo beat data-parallel as predicted while cutting total communication by more than 100× ². This 10B result — not any separate “DiLoCo-XL” paper — is the verified anchor for the claim that low-communication training holds at the 10B frontier.

The same line of work exposed a real weakness: DiLoCo degrades as the number of workers K grows. The 2025–2026 literature is largely a series of attacks on that degradation from three directions. Inner optimizer: Muon as the inner optimizer produces more directionally correct pseudo-gradients than AdamW as K increases, improving DiLoCo exactly in the regime where it degrades ⁵. Aggregation: an explicit analogy between DiLoCo aggregation and model merging asks whether the rich merging literature (task vectors, etc.) can replace naive averaging when many workers sync after long local horizons ¹². Outer optimizer: primal-averaging generalizations of Nesterov’s method unify single-worker DiLoCo with Schedule-Free-style averaging and reach targets in ~9–10% fewer steps than AdamW ⁶; and staleness-aware outer optimizers gate pseudo-gradients by age for asynchronous settings ⁸. The shared lesson: DiLoCo’s behaviour is governed by the pseudo-gradient quality into the outer loop, which is exactly what the local-SGD theory line said would matter ¹⁵.

Theme 4 — Unblocking the outer loop: streaming, eager updates, gossip, desync

Even at 100–500× less communication, a synchronous DiLoCo outer step still blocks: all workers stop local training, exchange, and wait for the straggler. A fast-moving 2025–2026 variant literature removes that stall in several complementary ways.

Streaming DiLoCo overlaps communication with computation by streaming partial parameter updates rather than exchanging one monolithic snapshot, and compresses the outer gradients to low precision — eliminating the peak-bandwidth spike of each sync ³. Eager updates let the shared (global) model advance optimistically while outer gradients are still in flight ⁴. Factored gossip splits the sync into a non-blocking mixing step that overlaps compute with no staleness, plus a blocking step that tightens agreement — a tunable dial between utilization and stability ¹³. GASLoC pushes the same idea to its logical end: gossip-based decentralized training that generalizes “communication acceleration” to the outer optimizer and works with adaptive optimizers and sparse randomized peer communication ²². NoLoCo removes even the residual all-reduce over model replicas that other low-communication methods keep ¹⁰. DES-LOC assigns different synchronization periods to parameters versus optimizer momenta — cutting communication 170× versus DDP and ~2× versus prior Local-Adam while keeping convergence guarantees and tolerating system failures ⁹. DeMo attacks the same wall from the optimizer side: decouple momentum, sparsify with a fast transform, reuse the momentum buffer as error feedback — up to 85× less data per GPU than AdamW on 300M–1B models ¹⁴. In asynchronous settings, staleness itself becomes the enemy: HeLoCo adds direction-aware corrections for stale pseudo-gradients, explicitly covering the non-IID data regime that asynchronous low-communication training had ignored ¹¹.

The most radical unblocking is architectural: Decoupled DiLoCo abandons the shared-model lockstep entirely ⁷. Independent “learners” run local inner loops and communicate only their parameter fragments to a central synchronizer, which aggregates with a minimum quorum, an adaptive grace window, and dynamic token-weighted merging — so failed or straggling learners are simply excluded until they return (recovery pulls state from a peer). In chaos-engineering simulations over millions of chips with continuous failures, training maintains strictly zero global downtime while preserving quality on text and vision, dense and MoE ⁷. This is the paper where the two halves of this review start to be one subject.

Theme 5 — It actually runs: multi-continent and internet-scale training

The strongest evidence that low-communication training is not a lab curiosity comes from runs that happened outside data centers. OpenDiLoCo reimplemented DiLoCo open-source on the Hivemind peer-to-peer stack and trained across two continents and three countries at 90–95% compute utilization, with FP16 all-reduce of outer gradients costing nothing — and scaled to 3× the original work ²³. INTELLECT-1 went further: the first 10B-parameter model trained collaboratively across the globe — 1 trillion tokens, up to 14 concurrent nodes on 3 continents, 30 independent compute providers joining and leaving, 83–96% compute utilization, and a 400× communication-bandwidth reduction versus data-parallel via its PRIME framework (ElasticDeviceMesh for dynamic process groups, live checkpoint-recovery kernels, hybrid DiLoCo-FSDP2) ²⁴. INTELLECT-2 extended the paradigm to fully asynchronous reinforcement learning of a 32B reasoning model over a permissionless, heterogeneous swarm, adding verification of untrusted rollout workers and policy-weight broadcast ²⁵.

Agora/Pluralis-8B is the current frontier of the community-driven line: an 8.6B model on 500B tokens trained over 40 days by ~330 contributor nodes — mostly consumer GPUs on ordinary internet connections — at ~170k tokens/s and 4.2 tokens per TFLOP, about 63% of a centralized H100 baseline’s efficiency, with no single party ever holding the full weights ²⁶. On the organizational side, DiLoCoX targets >100B-parameter decentralized training with pipeline parallelism, a dual-optimizer policy, one-step-delay overlap and adaptive gradient compression ²⁷; Atlas trains over WAN with workload-aware temporal bandwidth sharing and fills residual bubbles with prefill-as-a-service ²⁸; HALoS uses hierarchical local-plus-global parameter servers to minimize inter-region traffic with convergence guarantees ²⁹. The field now has its own survey, which categorizes exactly this split between community-driven and organizational approaches ⁴⁶.

Theme 6 — Geography is an energy story: curtailment, power-aware clusters, fiber

For AI-factory economics the point of decoupling training from one site is not just resilience — it is electricity. A 2026 feasibility study trained a 561M-parameter model across geo-distributed clusters scheduled into renewable curtailment windows (times when grids waste clean power), elastically switching between local single-site training and federated multi-site synchronization as sites appear and disappear; preliminary results show operational emissions dropping to 5–12% of single-site baselines at preserved quality ³⁰. PowerScale generalizes the insight: form training clusters by both network proximity and power availability, and a 100-site simulation matches time-to-accuracy while cutting energy up to 3.9× versus single-tier WAN aggregation — whose synchronization barriers idle faster sites and whose full-WAN updates dominate energy ³¹. Even the fiber matters: discrete-event simulation puts the optimum distance between two AI clusters at 10–100 km, where hollow-core fiber buys ~25% more compute-communication overlap ³². None of these are deployed-fleet results — all are simulations or small prototypes — but they are the only quantitative bridge from “DiLoCo works” to “training can follow cheap, clean, stranded power,” which is the version of this literature that matters for datacenter siting decisions.

Theme 7 — The checkpoint tax, quantified

The case for checkpointless recovery rests on how bad checkpoint-restart is at scale. The most cited evidence is operational: at 10,000+ GPUs, failures and stragglers are normal, and MegaScale’s production system treats fast checkpointing and recovery as a first-class co-design problem — the bar being to catch up within ~15 minutes and hold ≥90% effective training time ³³. Measurement studies add texture: checkpoint-restart costs differ dramatically across frameworks, and naive checkpointing exposes serious I/O bottlenecks ⁴²; the operations themselves consume measurable energy, modulated by processor power states and problem size ⁴³; and checkpoint storage has grown so large that lossy compression with training-time-adaptive sensitivity — delta-encoding consecutive checkpoints — is considered acceptable ⁴⁴. Hardware reality amplifies all of it: the companion review of GPU memory error rates in this library documents the bursty, cluster-to-cluster variability that makes failures hard to predict ([/reviews/ecc-memory-errors/]). The frequency-vs-cost dilemma is the crux: checkpoint rarely and a failure rolls back hours; checkpoint often and the I/O tax eats the run. Every system in the next theme is a different answer to that dilemma.

Theme 8 — The mitigation stack: every way to not do full checkpoint-restart

The academic response to the checkpoint tax is a spectrum of mechanisms, none of which (until very recently) called itself “checkpointless”:

Cheaper persistence. DataStates-LLM decouples training state from the process with lazy, asynchronous checkpointing over composable state providers, cutting the I/O cost of frequent saves ³⁵; ByteCheckpoint unifies save/load and — critically — resharding across parallelism configurations, so a checkpoint is usable by the next job, not just the one that wrote it ³⁶.

Faster detection and recovery. FlashRecovery adds continuous active failure detection (seconds-scale) feeding fast low-cost recovery ³⁷. FFTrainer leverages surplus network capacity to save and load state so frequently that rollback nearly disappears — up to 98% recovery-time reduction and 68% less GPU-utilization loss versus prior checkpointing ³⁸. PHOENIX hot-swaps failed nodes with spares along a zero-overhead checkpoint path, attacking the steady-state-overhead versus recovery-latency Pareto frontier directly ³⁹.

No checkpoint at all for reconfiguration. ETC migrates model state directly between device sets when elasticity (preemption, scaling) changes the parallel configuration — explicitly checkpoint-free, exploiting state locality instead of persisting and reloading everything ⁴⁰. Bamboo showed earlier that even the extreme failure model of cheap preemptible instances can be trained through, via partial re-training and redundancy ⁴¹.

Replicated state instead of saved state. Oobleck instantiates f+1 logically equivalent pipeline replicas so any f simultaneous failures recover from already-replicated in-memory state — no checkpoint-restart, no resource idling — with a formal guarantee that surviving templates cover all resources ³⁴. This “keep a live copy of the state somewhere else” idea is the direct ancestor of both torchft and AWS checkpointless training.

The production frameworks ship these as integrated stacks: TorchTitan (Meta’s PyTorch-native pretraining system) integrates asynchronous checkpointing and fault-tolerant HSDP as named components ⁴⁵.

Theme 9 — “Checkpointless training” as shipped: AWS HyperPod, torchft, and what the words mean

In December 2025 the term entered product vocabulary: AWS announced checkpointless training on SageMaker HyperPod, alongside elastic training (AWS News Blog, 2025-12-03 — vendor-stated). The launch post describes it as eliminating “disruptive checkpoint-restart cycles, maintaining forward training momentum despite failures, reducing recovery time from hours to minutes,” built from four components: collective-communication initialization optimizations, memory-mapped data loading with caching, in-process recovery, and checkpointless peer-to-peer state replication, orchestrated by the HyperPod training operator. AWS states that Nova models were trained with this technology on tens of thousands of accelerators, and that internal studies across 16-to-2,000+ GPU clusters measured downtime reductions over 80% versus checkpoint-based recovery — figures from internal studies that cannot be independently verified (launch post, docs, recipes incl. GPT-OSS 120B fine-tune). The mechanism — redundant model state held in GPU memory across peers, recovered in-process without restart — is the replicated-state idea from Oobleck ³⁴ and from Meta’s torchft library, which implements fault tolerance for HSDP/DDP/LocalSGD and DiLoCo (marked experimental) with “live checkpoint recovery from a healthy” replica (torchft GitHub; repo-documented, no paper or DOI exists).

Two framing points matter. First, “checkpointless” here means no checkpoint-restart on failure — it does not mean no checkpoints anywhere: evaluation, experiments, and stage transitions still need saved states, and AWS’s own stack saves optimizer state for elasticity. Second, none of the shipped systems are DiLoCo: AWS checkpointless training targets ordinary synchronous HSDP workloads, and torchft’s DiLoCo support is the only shipped bridge between the two lines — which is why the research-side combination below matters.

Theme 10 — The combination: resilient DiLoCo as the frontier

The two lines combine most completely in Decoupled DiLoCo ⁷, whose design reads like a checklist of both literatures: low-communication DiLoCo inner/outer structure (500×-class bandwidth reduction), decoupled momentum-style optimization ¹⁴, asynchronous staleness management that later work refines with age-aware outer optimizers ⁸, quorum-based aggregation that simply outlasts failed learners, peer-to-peer state recovery on rejoin (the torchft/Oobleck move), and — tellingly — internal Chandy-Lamport distributed snapshots plus deterministic-replay logging for state coordination, i.e. checkpoints as a coordination primitive rather than a recovery primitive. Google DeepMind’s own framing of the work is explicitly about resilience at scale (Decoupled DiLoCo blog, 2026-04-23).

The pattern across INTELLECT-1 ²⁴, Agora ²⁶, and Decoupled DiLoCo ⁷ is that churn is no longer an exception to engineer around but the design condition: training proceeds through nodes leaving, failing, or being scavenged, and quality holds. This is exactly the property a checkpointless regime needs, because when failures stop stalling the run, the rate of checkpointing becomes a choice about experiments and reproducibility rather than survival. What is missing is a shipped, at-scale system that is simultaneously low-communication and checkpointless for ordinary (non-research) workloads — torchft is experimental, AWS’s product is synchronous-only, and the research systems run in simulation or at community scale.

Where the evidence disagrees

Local SGD theory vs. DiLoCo practice. The theory line says local SGD does not reliably beat minibatch SGD and heterogeneity hurts ¹⁸¹⁹, and a production-scale study found accuracy losses that grow with scale ²⁰. DiLoCo’s results ¹² sit in apparent contradiction. The resolution in the literature is that DiLoCo is not naive local SGD: the outer optimizer (Nesterov momentum over pseudo-gradients, later analysed directly ¹⁵) changes the regime, and heterogeneity is controlled because each worker trains on large IID-ish shards rather than federated edge data. The disagreement is best read as “the naive algorithm underperforms; the two-loop design is what rescues it” — but the burden of proof for very large K still sits with the DiLoCo camp ⁵¹².

Synchronous vs. asynchronous. The synchronous DiLoCo papers emphasize predictable convergence; the async camp argues blocking is the real tax ⁷⁴, while staleness critics note async updates can be misaligned with the global direction — especially non-IID ¹¹ — and propose gating ⁸. No head-to-head at equal scale settles it.

Vendor claims vs. independent evidence. AWS’s >80% downtime reduction and “Nova on tens of thousands of accelerators” are internal-study claims [AWS launch post]; INTELLECT-1’s quality claim (“comparable performance”) is a single community run ²⁴; Agora’s 63% efficiency is one run ²⁶. These are consistent with the academic results but not yet independently reproduced at scale.

Naming. The “DiLoCo-XL” label, widely used in discussion for a 10B/two-continent result, matches no primary source; the verified 10B anchors are the scaling-law validation ² and INTELLECT-1 ²⁴. Reviewers and engineers repeating “DiLoCo-XL” should say which result they mean.

Gaps and open questions

• “Checkpointless” has no academic identity. Zero arXiv records use the term; the science is scattered across async checkpointing, replicated-state recovery, state migration and erasure coding. A survey or systems paper that names and bounds the design space (what must still be persisted, when, and why) would materially help.
• Frontier-scale low-communication training is untested. All verified DiLoCo-scale evidence stops at 10B (scaling-law validation and INTELLECT-1). Nothing public shows 100B+ or trillion-token low-communication training, which is where the scaling-law extrapolations ² point but do not prove.
• Decoupled DiLoCo’s zero-downtime claim is simulation-based (“millions of simulated chips”); real multi-site chaos deployments are the missing confirmation ⁷.
• Energy results are simulation-scale (curtailment feasibility on a 561M prototype ³⁰; PowerScale in a 100-site simulator ³¹). No published measurement of a real multi-site training run’s energy ledger exists.
• Checkpointless determinism: if training restarts from replicated state rather than a canonical checkpoint, what happens to reproducibility, experiment branching, and eval provenance? The Decoupled DiLoCo answer (snapshots as coordination state) is partial ⁷.
• Elastic + checkpointless + low-communication as one shipped stack (the torchft DiLoCo path [torchft GitHub] crossed with AWS-style peer-to-peer recovery) is the obvious product-shaped gap.

Confidence and limitations

Confidence is moderate overall. The mechanism-level claims (DiLoCo cuts communication by 100–500× at parity in the tested regimes; recovery without full checkpoint-restart is feasible and shipped) are well-supported by consistent, fetched primary sources, several read in full text. Confidence is lower for: frontier-scale extrapolation (10B is the verified ceiling), all vendor/internal-study numbers (AWS, and to a lesser degree community-run self-reports), and anything energy-related (simulation-based). Limitations of this review: five papers were read in full; the rest rest on fetched abstracts, so fine-grained claims from bodies may be missed; industry material (AWS, torchft, DeepMind blog) is vendor-documented and flagged as such; two of three delegated research waves timed out mid-run and were reconstructed from their partial files and this session’s own verification fetches; and the field is moving quickly enough that 2026 papers appearing after 2026-09-06 will not be reflected. Searches were in English; venues were arXiv-first, which is where this literature lives.