---
title: "Optimizing MLPerf Training closed-division submissions: what the literature shows"
slug: mlperf-training-closed-optimization
question: Which methods documented in the literature most reduce time-to-quality for MLPerf Training closed-division submissions, given the rule that model, optimizer and most hyperparameters are fixed?
status: published
depth: standard
created: 2026-08-07
updated: 2026-08-07
summary: "A review of 47 sources on how to minimize time-to-quality in the MLPerf Training closed division, where the model, optimizer, and most hyperparameters are fixed by the rules. The levers the evidence supports are: picking the global batch size that sits at the reference-convergence boundary, reduced-precision training (bf16/fp8, now also fp4), distributed-parallelism and communication-overlap engineering, data-pipeline and kernel work, and benchmark-specific plays for DLRM, LoRA fine-tuning, and MoE pretraining. Each lever is worth roughly 1.2–2× in isolation and they compound, but the convergence-speed cap (Reference Convergence Points) means the winning move is maximum throughput at the RCP boundary, not faster convergence. Confidence is moderate: the per-lever results are replicated systems literature, but no public study directly tests the full closed-division strategy."
disciplines: ["computer systems", "machine learning"]
tags: ["MLPerf", "distributed training", "mixed precision", "large-batch training", "communication overlap", "HPC"]
source_count: 47
year_range: [2016, 2026]
confidence: moderate
search:
  databases: ["OpenAlex", "Crossref", "arXiv", "Semantic Scholar"]
  queries:
    - "MLPerf training benchmark"
    - "MLPerf training performance analysis submission"
    - "large batch training convergence deep learning"
    - "mixed precision training deep neural networks"
    - "FP8 training large language models"
    - "distributed training communication overlap GPU cluster"
    - "pipeline parallelism language model training"
    - "data loading deep learning training performance bottleneck"
    - "activation checkpointing memory efficient training"
    - "stochastic rounding numerical precision machine learning"
    - "mixture of experts large language model training scalability"
    - "DLRM recommendation model training optimization"
    - "LoRA low rank adaptation fine-tuning large language models"
    - "FlashAttention fast memory efficient attention"
    - "DeepSeek-V3 training"
    - "ZeRO memory optimization distributed training"
    - "bfloat16 deep learning training"
    - "TensorFloat-32 tensor core precision"
    - "GPU utilization training throughput optimization"
    - "large language model pretraining efficiency Chinchilla"
    - "targeted title-resolution of 50 known-candidate papers via Crossref bibliographic + OpenAlex title.search"
  last_run: 2026-08-07
---


## Summary

The MLPerf Training closed division fixes the model, the optimizer, and most hyperparameters, so the score is a race to a fixed quality target under a convergence-speed cap. The literature says the race is won on four axes: choosing the global batch size that lands at the boundary of the allowed convergence range, training in the lowest pre-approved precision (bf16/fp8, with fp4 emerging), engineering the distributed system (parallelism, communication overlap, data pipeline, kernels) so each epoch is as cheap as possible, and exploiting benchmark-specific structure (embeddings for DLRM, adapters for LoRA, expert routing for MoE). Individual techniques are worth roughly 1.2–2× each and compound. The most important caveat: the rules' Reference Convergence Points cap how much faster than the reference you may converge, and a submission that converges too fast is normalized to the reference's epoch count — so the optimal strategy is maximum throughput at the RCP boundary, and the literature on that specific optimization is essentially nonexistent; it must be inferred from rules plus adjacent systems results.

## Why this question

MLPerf Training is the only widely recognized benchmark where a closed-division submission must reproduce a reference model's exact graph, initialization, and quality target while still racing the clock. Submitters (NVIDIA, Google, and a handful of others) publish results but not playbooks; the rules document defines what is legal but not what is fast. Anyone preparing a submission — including the author of this review — needs the transferable evidence: which published techniques are (a) effective at reducing wall-clock time to quality and (b) still inside the closed division's equivalence rules.

What turns on the answer is real money and engineering time: a v6.0-scale submission (Llama 3.1 405B or DeepSeek-V3 671B pretraining) costs millions of GPU-hours, and a wrong assumption about what the rules permit — say, adopting a communication-compression scheme that changes the optimizer's math — can invalidate a submission at review time. The review is therefore as much a compliance map as a performance survey.

## Scope and methods

**Question.** Which methods documented in the literature reduce time-to-quality for MLPerf Training closed-division submissions, given fixed model, optimizer, and most hyperparameters?

**Population.** MLPerf Training closed-division benchmarks across versions (ResNet-50, BERT, GPT-3, Llama 3.1 8B/405B, GPT-OSS-20B, DeepSeek-V3-671B, DLRMv2, Llama2-70B-LoRA, FLUX.1). **Concepts.** Large-batch training and learning-rate schedules; mixed and reduced precision (fp16/bf16/tf32/fp8/fp4, stochastic rounding); memory optimization and parallelism (data/tensor/pipeline/expert); communication optimization and overlap; data loading; kernel-level optimization; benchmark-specific engineering. **Outcomes.** Wall-clock time to quality target, epochs-to-convergence, throughput, speedup ratios. **Inclusion.** Peer-reviewed systems/ML papers, arXiv preprints, and benchmark-definition documents from 2015–2026, English. **Exclusion.** Inference benchmarks, MLPerf HPC/Storage/Inference variants (one storage characterization paper kept for the I/O argument), open-division model changes, gradient-compression as a recommended lever (flagged where relevant), hardware microarchitecture papers without training results.

**Search and screening.** 20 layered queries (core-concept + synonym + filter blocks) across OpenAlex, Crossref, arXiv, and Semantic Scholar on 2026-08-07 (Semantic Scholar was rate-limited and returned nothing usable; OpenAlex's budget exhausted mid-run and was completed from a second IP). Raw retrieval 2,594 records; 2,350 unique after DOI/title dedup; 2,291 after topical exclusion of off-topic hits. A targeted title-resolution pass over 50 known-candidate papers (Crossref bibliographic + OpenAlex title.search) recovered the canonical literature that keyword search missed; three of my own remembered arXiv IDs resolved to different papers and were discarded after abs-page verification. Full-text screening and evidence extraction included 47 sources (30 read in full text, 17 abstract-only). Every DOI was verified against Crossref or DataCite this session; no source is retracted. Numbers cited here were spot-checked against full texts by grep; unverifiable figures were dropped.

## The landscape

The literature splits into two largely disconnected bodies. The first is the **systems literature** (SC, ISCA, ASPLOS, MLSys, VLDB): how to make a fixed training workload run fast at scale — parallelism, communication, memory, I/O. The second is the **optimization/ML literature** (ICLR, NeurIPS, arXiv): how many steps a given model needs to reach quality, and which hyperparameters control that. MLPerf-specific work is tiny: the benchmark definition paper [1], one published case study of a record BERT submission [2], one storage benchmark [42], and a handful of cost-prediction/energy papers. Almost everything a submitter needs must be assembled from adjacent fields.

The corpus is dominated by three institutions — NVIDIA (mixed precision, Megatron, FlashAttention, network optimization), Microsoft (ZeRO, DeepSpeed, data stalls, 1-bit LAMB), and Google (GPipe, GShard, Switch, Chinchilla) — so "consensus" in places is one lab's program replicated by others. The field is young: the oldest load-bearing result here is 2016, and the median is 2021–2022, meaning most evidence predates the current v6.0 benchmark suite (Llama 3.1 405B, DeepSeek-V3 671B, FLUX.1). One era dominates: the large-batch/optimizer debate (2016–2021) and the extreme-scale systems work (2020–2024) are mature; the fp4-era and v6-era submission analysis are open territory.

The rules frame everything that follows, so they are quoted here as the constraint set: the [MLPerf Training Rules v6.0](https://github.com/mlcommons/training_policies/blob/master/training_rules.adoc) require the same preprocessing, model graph, initialization, loss, and quality target as the reference; hyperparameters are fixed except a short tunable list; the optimizer implementation must match the reference; and only numerically safe precision-compensation techniques are pre-approved (scaling, orthogonal transforms like Hadamard, stochastic rounding). Everything else in this review is read against that boundary.

## What the closed division actually fixes

The benchmark definition paper makes the central design explicit: MLPerf Training measures time to a quality target, not throughput, and the closed division enforces equivalence to the reference implementation precisely so that faster hardware and engineering — not clever model changes — are what win [1]. That framing has three consequences that shape every technique below.

First, **convergence speed is capped, not rewarded**. The rules require submissions to match Reference Convergence Points (RCPs): a one-sided t-test against reference convergence data allows only ~3.5–4% faster mean epochs-to-convergence (the rules' worked example), and any submission converging faster is normalized to the RCP mean. The MLPerf paper documents the underlying phenomenon — epochs-to-target grows with batch size, e.g. ResNet-50 needing ~64 epochs at 4K batch versus more than 80 at 16K, a 30% compute increase [1]. So the score is effectively *(RCP-mean epochs) × (time per epoch)*, and the first-order lever is the batch size you pick, because it fixes the epoch budget you are allowed.

Second, **the clock is narrower than it looks**. Model initialization (graph compile, checkpoint load) is untimed up to 30 minutes in closed division; only per-run preprocessing, training, and quality evaluation are on the clock; and for benchmarks with timestamped checkpoints, evaluation can run off-clock with the stop time rolled back to the first passing checkpoint. Nothing in the reviewed literature studies this explicitly, but the rules themselves note that coarse evaluation spacing cost v5.0-era Llama 405B submissions a 12–16% overhead when convergence landed just after an evaluation, which is why v6.0 evaluation frequencies were tightened.

Third, **the optimizer is fixed but its implementation is a differentiator**. The rules name the optimizer per benchmark (AdamW for the LLM benchmarks, Adagrad for DLRMv2) and require the implementation to match the approved list, with alternate implementations needing an equivalence demonstration. Within that, the case-study literature shows how much implementation quality matters: the record BERT submission re-tuned Adam (LR, betas, weight decay, and gradient-clip placement) inside the closed rules and beat the previous best closed submissions by 1.33× (v1.1) and 1.57× (v2.0) [2].

## Batch size: the highest-leverage tunable

The global batch size is the one hyperparameter the rules leave essentially unconstrained for most benchmarks, and it is the one with the most direct evidence of wall-clock effect. The canonical result is Goyal et al.: with linear learning-rate scaling and gradual warmup, ResNet-50 trains at an 8k batch in one hour on 256 GPUs with accuracy statistically indistinguishable from the small-batch baseline (23.74% vs 23.60% top-1 error) [3]. The warmup detail matters — without it the 8k run degrades by over a point — and the same warmup-plus-scaling pattern recurs in every later large-batch recipe.

The early literature warned that large batches generalize worse: Keskar et al. measured gaps up to 5% and attributed them to convergence to sharp minima [4]. That claim is contested (see below), and the practical resolution came from the optimizer line: LAMB's layerwise-adaptive rates enabled BERT pretraining at batch 32,868, cutting time from 3 days to 76 minutes on a TPUv3 Pod [5], and a careful 2021 study under the actual MLPerf ResNet-50 rules showed that plain Nesterov momentum reaches the 75.9% target in 2,512 steps at batch 32,768 — matching the best published LARS result — while tuned Adam beats LAMB on BERT at batch 65,536 (F1 90.46 in 7,818 steps) [6]. The generalizable finding is that the optimizer's identity matters less than having one that tolerates the batch size, and that the "special large-batch optimizers" narrative overstated the gap.

For picking the batch size itself, the modern evidence is the critical-batch-size (CBS) literature: CBS starts near zero, rises quickly, then plateaus, and batch-size warmup — start small, grow with CBS — trained OLMo 1B to slightly better loss with 43% fewer gradient steps [7]. The rules-compatible reading is awkward: closed-division runs must hold a single GBS for RCP matching, so the CBS result is a pointer toward *which fixed* GBS to choose (roughly the plateau value for the target loss), not a license to schedule batch growth mid-run — a mapping the literature has not tested. Chinchilla's compute-optimality result (model size and tokens should scale equally) [8] is background context: the v6.0 LLM benchmarks are deliberately undertrained relative to compute-optimal, which is precisely why their RCPs allow hitting the target in a bounded token budget.

## Precision: throughput that costs nothing

Reduced precision is the cleanest lever in the whole review because the rules pre-approve the formats and the evidence shows accuracy parity at unchanged hyperparameters. The foundation is the mixed-precision recipe — fp16 storage with fp32 master weights, loss scaling, fp32 accumulation — which matched fp32 accuracy without retuning across vision, speech, and language workloads [9], with bf16 achieving the same parity while keeping fp32's dynamic range and thus requiring zero hyperparameter changes [10]. The hardware side was established by the Tensor Core programmability work: mixed-precision tensor cores deliver roughly 7× single-precision throughput on V100-class hardware [11], which is why every modern submission is precision-first.

The current frontier is fp8: the format definitions paper showed fp8 (E4M3/E5M2) training matches fp16/bf16 quality up to 175B parameters with unchanged hyperparameters [12], and the FP8-LM system made it concrete — GPT-175B trains 75% faster than the bf16 Megatron-LM baseline with 39% less memory [13]. DeepSeek-V3 validated fp8 at extreme scale in production: 671B parameters, 2.788M H800 GPU-hours total (~$5.6M at $2/GPU-hour), with fp8 mixed precision as a core co-design [30]. Below fp8, the first fp4 training framework reports accuracy comparable to bf16/fp8 at up to 13B parameters and 100B tokens [14] — notable because the v6.0 rules already pre-approve nvfp4/mxfp4. A 2026 survey consolidates the space: fixed-point, floating-point, and custom formats, with fp8/fp4 native paths on Hopper/Blackwell [17].

The numerically safe compensation techniques the rules explicitly allow are the subject of their own literature. Stochastic rounding — cited by name in the rules — has error bounds of order √n·u with high probability versus n·u worst case for round-to-nearest, and is immune to stagnation, the loss of tiny updates to large accumulators [15]. Systems work shows the payoff: variable-precision block floating point with stochastic rounding delivers 2–6× training speedups at similar accuracy [16]. The boundary to respect is the rules' own: scaling and rounding are allowed; anything that changes the model's real-valued function (e.g., altered normalization or gradient clipping) is not.

## Memory and parallelism: fitting the model on the machine

For the large-model benchmarks (405B dense, 671B MoE), the first question is whether the model fits at all, and the answer is a 2020–2022 stack: ZeRO partitions optimizer state, gradients, and parameters across data-parallel ranks, training 100B-parameter models on 400 V100s at 38 TFLOPS/GPU with super-linear speedup [18]; activation recomputation reduces activation memory to O(√n) at the cost of one extra forward pass — a 1,000-layer ResNet drops from 48 GB to 7 GB [19], and the rules' utilization appendix explicitly counts recomputation flops as non-model flops, signaling it is expected. Tensor parallelism (Megatron-LM) carries models that data parallelism cannot: 8.3B parameters on 512 GPUs at 15.1 sustained Petaflops with 76% scaling efficiency [20].

The composition playbook is Megatron's SC21 paper: pipeline + tensor + data parallelism with an interleaved schedule trains a 1-trillion-parameter GPT at 502 petaFLOP/s on 3072 A100s (52% of theoretical peak, 163 TFLOPS/GPU end-to-end), with the interleaved schedule alone worth up to 10% and microbatch count up to 15% [21]. The pipeline lineage that makes this possible: GPipe's batch-splitting with near-linear speedup [22], PipeDream's parameter versioning for numerically correct gradients [23], and TeraPipe's token-level pipelines for autoregressive models (5.0× on GPT-3 175B) [24]. The cost of scale is documented in the 530B run: per-GPU efficiency falls from 126 to 113 TFLOPS/GPU as the cluster grows from 280 to 420 DGX A100 servers [25] — the same scaling tax the closed division inherits, since the reference implementations (NeMo, Megatron-Bridge) define the parallelism baseline a submission must beat with engineering, not architecture. The survey literature [26] provides the organizing taxonomy for all of this.

For the MoE benchmarks (GPT-OSS-20B, DeepSeek-V3-671B), the sparse-axis evidence is strong: GShard trained a 600B-parameter MoE on 2048 TPUv3s in 4 days (22 core-years) versus 235.5 core-years for a dense 2.3B baseline [27]; Switch's single-expert routing gave up to 7× pretraining speedup at fixed FLOPs [28]; and DeepSeek-V3 contributes the modern system stack — DualPipe for near-full computation-communication overlap across forward/backward chunks, auxiliary-loss-free load balancing, and fp8 at scale [30]. The open systems problem is expert communication: shared-memory fusion of MoE communication and computation is the current frontier, with the argument that coarse-grained overlap impairs compute efficiency [29].

## Communication: the scaling tax

Once the model fits and precision is low, wall-clock time is dominated by how much of the interconnect's cost can be hidden. The end-to-end ceiling was set early: a communication backend combining ring allreduce, mixed precision, overlap, and lazy allreduce trained ImageNet/AlexNet 95 epochs in 1.5 minutes on 512 GPUs (410.2×) and ResNet-50 in 7.3 minutes (434.1×) [31] — on 56 Gbps-era networking, which is the point: communication software was the bottleneck, not the hardware.

The modern evidence refines this into three moves. **Dedicated collective hardware**: ACE, a collective-communication engine at the accelerator endpoint, cuts the memory bandwidth needed to drive the network by 3.5× and raises effective network bandwidth utilization by 1.44× (up to 2.67×) [32]. **Software overlap**: CO2's local-updating plus asynchronous communication achieves full overlap and a scalability ratio of 1.08 from 16 to 128 GPUs versus AdamW's 0.93 [34]; FLUX fuses over-decomposed communication and compute kernels to overlap up to 96% of tensor-parallel communication, yielding up to 1.24× training speedup over Megatron-LM [35]; decomposition-based overlap for intra-layer model parallelism addresses the same problem at the schedule level [33]; and communication partitioning enables overlap in hybrid-parallel settings where fusion or op-scheduling alone fails [37].

The honest counterweight is a 2025 characterization study: overlapping communication with compute causes an average 18.9% (up to 40%) compute slowdown versus non-interfered execution, and power capping severely exacerbates it — though overlap still nets an average 10.2% (max 26.6%) over sequential execution [36]. Overlap is not free; it is a trade whose value depends on hardware, precision, and power state. The record BERT case study shows the highest-value overlap trick inside the closed rules: gradient clipping *before* allreduce (bucket-wise) so synchronization overlaps gradient computation, rather than after [2]. One caution: compressed-communication systems like 1-bit LAMB [38] are effective on low-bandwidth clusters but change the optimizer's real-valued function via error feedback, and would need explicit working-group approval under the closed division's numerical-equivalence rule — treat as non-compliant unless approved.

## Data pipeline: the hidden third of wallclock time

The systems literature's most repeated finding is that training is often stalled waiting for data. A Microsoft production study across nine models found training time frequently dominated by data-stall time — waiting for fetch and preprocessing [39]; locality-aware loading with software caches gives >30× data-loading speedup at 1,024 learners [41]; and sharing one data loader across collocated processes raises throughput up to 100% and cuts cloud CPU cost by half [40]. The MLPerf community's own storage benchmark was motivated by the same observation: datasets are outgrowing main memory, and I/O is the neglected axis [42].

The closed division makes this literature directly actionable because the rules draw a clean line: the dataset may be reformatted and preprocessed off-clock (only per-run random transforms are timed), but it must start on durable storage — explicitly not RAM — and caches must be flushed before a run. So the compliant data play is: aggressive offline preprocessing, sharded and packed formats, fast NVMe, and a pipeline deep enough that on-clock stalls vanish. The BERT case study quantifies the ceiling of one such trick: sequence-length packing (allowed as arbitrary batching, subject to RCP approval) has a theoretical 2× speedup available from load balancing alone, because unpadded BERT at length 512 wastes nearly half its compute on padding [2].

## Kernels and graphs: mathematically equivalent speed

The closed division explicitly permits graph transformations that are mathematically equivalent — dead-code elimination, common-subexpression elimination, recomputation — and the strongest example in the literature is attention. FlashAttention's IO-aware tiling computes exact attention with linear memory and delivered a 15% end-to-end BERT-large speedup versus the MLPerf 1.1 training record, 3× on GPT-2, and up to 7.6× on the attention kernel itself [43]; FlashAttention-2's parallelism rework reaches ~2× beyond that, up to 225 TFLOPS/s per A100 (50–73% of peak) [44]. Because the math is unchanged, these kernels are unambiguously inside the closed division — they are the template for what "same graph, faster execution" looks like, and the reference implementations of the v6.0 LLM benchmarks already consume them.

## Benchmark-specific playbooks

**BERT (v5.0, deprecated).** The only fully documented closed-division optimization campaign: stratification-based presorting of the dataset plus bucket-wise clip-before-allreduce plus re-tuned Adam produced 25.1 s (22.3 s best run) on 1,024 A100s, 1.33×/1.57× faster than the top v1.1/v2.0 submissions [2]. Its transferable lessons: dataset-order engineering is legal (training data order must be "conceptually the same," and stratification preserves the random-order contract), evaluation-adjacent scheduling matters, and network topology (200 vs 100 GB/s, SHARP presence) explained a 43.5 s vs 59.5 s gap on nominally identical GPUs.

**DLRMv2 (v6.0).** The model is embedding-dominated: tables of multiple GB each make data parallelism prohibitive, forcing model-parallel embedding sharding with personalized all-to-all after lookup [45]. The SC24 work adds the modern optimization set for the fixed-Adagrad closed benchmark: tensor-train decomposition of embeddings plus sorted-index micro-batching plus pipelined embedding-layer training to cut both compute and communication [46]. Since DLRMv2 trains at most one epoch and the LR schedule is fixed, this benchmark is the purest throughput race in the suite.

**Llama2-70B-LoRA (v6.0).** Fine-tuning with frozen weights and rank-decomposition adapters cuts trainable parameters 10,000×, GPU memory 3× (1.2 TB to 350 GB on GPT-3 175B), and gives a 25% training speedup because gradients for the vast majority of parameters are never computed [47]. For a benchmark whose reference already fixes LoRA alpha/rank/dropout, the remaining levers are the system ones: the model is small enough to data-parallel with aggressive overlap, and the evaluation schedule (every 384 sequences) rewards tight checkpointing.

**MoE pretraining (GPT-OSS-20B, DeepSeek-V3-671B).** Covered by the MoE and systems evidence above; the reference implementation (NeMo/Megatron-Bridge for DeepSeek) already embodies the DualPipe/fp8 stack [30], so submitters compete on the margins: expert-parallel communication fusion [29], load-balancing stability, and GBS choice within the fixed LR formula.

## Where the evidence disagrees

**Does large batch hurt generalization, and does it matter?** Keskar et al. reported gaps up to 5% and the sharp-minima explanation [4]; Goyal et al. showed no loss up to 8k with proper LR scaling and warmup [3]; and the MLPerf-rules study found generic optimizers close the gap at even larger batches [6]. The best resolution is chronological and methodological: the early result used fixed hyperparameters; once warmup and scaling were treated as first-class, the gap largely vanished — the sharp-minima story was an artifact of undertuned optimization, a position the later evidence supports.

**Is LAMB special?** You et al. reported Adam failing beyond batch 16,384 and LAMB fixing it [5]; Dahl et al. did not reproduce the failure, finding tuned Adam beats LAMB at batch 65,536 [6]. The disagreement is about tuning budget and metric (steps vs wall-clock): LAMB's advantage was real on the TPU stack of its era with the tuning performed, and not reproducible as a law. For closed division the practical consequence is mild — the optimizer is fixed by the rules anyway — but it changes how much to trust reference hyperparameters.

**Is communication overlap free?** CO2 and FLUX present overlap as near-total recovery of the communication tax [34] [35]; the ISPASS 2025 characterization measures a real compute cost (18.9% average, up to 40%) and finds the net gain shrinks to ~10% on average, with power capping turning it negative [36]. The disagreement dissolves once you separate measurement conditions: the negative side appears when compute and communication contend for the same execution resources (power-capped, fused-poorly), while the positive results come from systems engineered specifically to avoid contention (dedicated engines, kernel fusion, over-decomposition). The practical rule: overlap pays, but measure it on your exact hardware and power budget.

**How much does batch size cost in epochs?** The MLPerf paper's ResNet-50 data shows epochs-to-target rising sharply with GBS [1], while the critical-batch-size literature says the loss-per-token penalty only appears beyond the CBS plateau [7]. These are consistent once you separate the regime: the MLPerf numbers are at 4K–16K where CNN training already exceeded the CBS; the CBS plateau is where a well-chosen GBS should sit. The tension is real only in that the closed division forces a single fixed GBS while the CBS picture is intrinsically dynamic.

## Gaps and open questions

**No published RCP playbook.** The single most important strategic question — how to choose GBS and LR so that mean epochs-to-convergence lands just at the RCP boundary while throughput is maximized — appears in no retrieved paper. It is rules-level knowledge (the t-test normalization, interpolation between RCP points, score scaling) that the literature has not touched. What would settle it: an empirical study of epochs-to-target distributions across GBS for a v6.0 benchmark, which only the reference owners (or a well-funded submitter) can produce.

**No v6.0-era submission analysis in the open literature.** All published case studies are BERT-era (v1.1/v2.0) [2]. The Llama 3.1 405B, DeepSeek-V3, FLUX.1, and DLRMv2 rounds have no equivalent public dissection; the one recent systems account (AMD Instinct submissions) is paywalled and could not be retrieved. The field's knowledge of what actually wins the current suite is therefore largely grey.

**Compliance of emerging techniques is untested.** Batch-size warmup [7] and fp4 training [14] are both promising and both sit in a gray zone: the former against RCP matching (fixed GBS), the latter against the "numerically safe" definition at scale. No working-group decision or public test exists for either in a closed-division setting. Similarly, compressed communication [38] needs explicit approval that no submitter has documented.

**Evaluation quantization is documented only in the rules.** The v5.0 note that coarse evaluation spacing cost 12–16% when convergence landed just after an evaluation is a rulebook observation with no systems paper behind it. The interaction between evaluation frequency, GBS, and the timestamped-checkpoint stop rule is a measurable, publishable effect that would directly inform v6.0 scheduling.

## Confidence and limitations

Confidence in the per-lever effects is moderate-to-high: the large-batch, precision, parallelism, and overlap results are replicated across independent groups and hardware generations, and the load-bearing numbers were verified against full texts in this review. Confidence in the overall "optimal closed-division strategy" is lower, because no source directly tests it — the synthesis is an inference from the rules (retrieved and read in full) plus adjacent systems evidence. Limitations of this review: Semantic Scholar contributed nothing (rate-limited), so coverage leans on OpenAlex/Crossref/arXiv; five sources are abstract-only because of paywalls or bot-blocking (most load-bearing claims still rest on full texts); the v6.0-specific literature is thin by the field's own youth; and the review excludes non-English work and inference-benchmark transfer (which is a real but separate literature). One source with no retrievable abstract was dropped rather than cited on title alone.

## Evidence table

| key | year | design | sample | measure | finding | limitations | confidence |
| --- | --- | --- | --- | --- | --- | --- | --- |
| goyal2017largeminibatch | 2017 | empirical study | ResNet-50 on ImageNet, 256 GPUs (Caffe2, commodity hardware) | top-1 validation error; wall-clock time to accuracy | Linear LR scaling + gradual warmup makes accuracy invariant to minibatch size up to 8k, training ResNet-50 to baseline-matching accuracy (23.74% vs 23.60% error) in 1 hour on 256 GPUs | Results demonstrated up to 8k batch only; tuned on ImageNet/ResNet-50; scaling-efficiency claim is for their specific interconnect/software stack | high |
| keskar2016largebatch | 2016 | empirical study | MNIST/TIMIT/CIFAR-10/CIFAR-100 nets (F1,F2,C1–C4), ADAM, SB=256 vs LB=10% of training data | generalization gap (test accuracy SB vs LB) | Large-batch training converges to sharp minimizers and suffers a generalization gap of up to 5% even on small networks | Small models/datasets, ADAM-focused, accuracy not state-of-the-art; sharp-minima explanation later contested | high |
| you2020lamb | 2019 | empirical study | BERT-Large pre-training, SQuAD-v1 fine-tune, TPUv3 Pod | wall-clock time to F1; SQuAD F1 score | LAMB enables BERT batch sizes of 32,868 without degradation, cutting pre-training from 3 days to 76 minutes on a TPUv3 Pod | TPU-specific; tuned warmup/decay; claims later contested by Dahl et al. (2021); F1 measured after fine-tune | high |
| dahl2021optimizerreality | 2021 | empirical study | ResNet-50 v1.5/ImageNet (MLPerf rules) and BERT pre-training | steps/epochs to 75.9% top-1; F1 at fixed step budget | Standard Nesterov momentum and Adam match or beat LARS/LAMB at large batch sizes (Nesterov: 75.9% in 2,512 steps at batch 32,768; Adam: F1 90.46 in 7,818 steps at batch 65,536) | Matching shown in training steps (argued to imply faster wall-time, not measured on identical hardware); required extensive hyperparameter search; medians over 50 runs | high |
| hoang2025criticalbatch | 2025 | empirical study | OLMo 1B and 7B language models (up to ~419B tokens) | loss vs tokens; gradient steps to target loss | Measured critical batch size (CBS) starts near 0, rises rapidly, then plateaus; batch-size warmup trains OLMo 1B to slightly better loss with 43% fewer gradient steps | OLMo family only; loss-based (no downstream eval); direct-measurement method is new/empirical | high |
| hoffmann2022chinchilla | 2022 | empirical study | 400+ transformer LMs, 70M–16B+ params, 5–500B tokens; Chinchilla 70B | loss under fixed compute budget; downstream accuracy | For compute-optimal training, model size and training tokens should scale equally; Chinchilla (70B, 1.4T tokens) beats much larger models at the same compute budget | Pretraining-only analysis; ignores inference/compute-optimality of serving; empirical power-law fits | high |
| micikevicius2018mixedprecision | 2017 | empirical study | CNNs and RNNs (ImageNet, speech, translation, language modeling) on Volta GPUs | accuracy parity vs FP32; op-level speedup | FP16 storage with FP32 master weights, loss scaling and FP32 accumulation trains to FP32 accuracy without hyperparameter changes, nearly halving memory | Speedups hardware-dependent (2–6× on DeepBench ops); loss-scaling needed for some models; slight accuracy *gains* on speech attributed to regularization | high |
| kalamkar2019bfloat16 | 2019 | empirical study | Image classification, speech recognition, language modeling, generative networks, industrial recommendation | accuracy parity in same iteration count | BFLOAT16 achieves the same state-of-the-art results as FP32 in the same number of iterations with no hyperparameter changes | Software-emulated BF16 (no native HW at the time); no wall-clock speedup numbers reported; breadth over depth per task | high |
| markidis2018tensorcore | 2018 | benchmark study | NVIDIA Tesla V100 (Volta), 640 Tensor Cores; WMMA/CUTLASS/cuBLAS | delivered TFLOPS in mixed precision | Tensor Cores deliver up to 83 TFLOPS in mixed precision on V100 — 7× and 3× the single/half precision performance | (abstract) Full-text file is an unparsed PDF — extraction limited to abstract; vendor microbenchmarks, not end-to-end training | moderate |
| micikevicius2022fp8formats | 2022 | empirical study | CNNs, RNNs, Transformers incl. LLMs up to 175B params | accuracy parity vs FP16/BF16 baselines | FP8 (E4M3/E5M2) training matches FP16/BF16 results across architectures and sizes up to 175B parameters with unchanged hyperparameters | Requires per-tensor scaling/clipping of GEMM inputs; no wall-clock speedup numbers in extracted text (hardware-dependent) | high |
| peng2023fp8lm | 2023 | system paper | GPT-175B training on H100 GPUs | training speed (%), real memory usage | FP8 automatic mixed-precision framework trains GPT-175B 75% faster than BF16 Megatron-LM (and 37% faster than Transformer Engine) with 39% lower memory | H100-specific; comparison baselines are specific frameworks (Megatron-LM, Transformer Engine); incremental adoption levels | high |
| liu2025fp4training | 2025 | system paper | LLMs up to 13B parameters trained on up to 100B tokens | accuracy/loss parity vs BF16/FP8 | First FP4 LLM training framework reaches accuracy comparable to BF16/FP8 with minimal degradation, scaling to 13B params / 100B tokens | No native FP4 hardware (simulated); "minimal degradation" not zero; limited scale vs production LLMs | high |
| croci2022stochasticrounding | 2022 | survey | n/a (error analysis of floating-point inner products) | error bound constants | (abstract) Stochastic rounding gives inner-product error bound ~√n·u with high probability vs n·u worst case for round-to-nearest, and is immune to stagnation | (abstract) Abstract-only source; theoretical bounds, not end-to-end training results | moderate |
| kim2022fast | 2022 | system paper | DNNs on a single-chip platform with block floating point (BFP) arithmetic | training speedup; validation accuracy | (abstract) FAST system with variable-precision BFP gives 2–6× training speedup over prior mixed-precision/BFP systems with similar validation accuracy | (abstract) Abstract-only source; single-chip platform/simulation context; speedup vs prior work, not vs FP32 baseline | moderate |
| mei2026lowprecisionllm | 2026 | survey | Low-precision LLM training literature | taxonomy coverage (qualitative) | (abstract) Survey categorizes low-precision training into fixed-point/integer, floating-point, and customized-format methods, noting notable training-efficiency gains | (abstract) Full-text file is an unparsed PDF — extraction limited to abstract; no quantitative speedup figures in abstract | moderate |
| mattson2020mlperftraining | 2020 | benchmark study | MLPerf Training v0.5 suite (ResNet-50, GNMT, Transformer, DLRM, Mini-Go, etc.), reference implementations | epochs to quality target; time to quality | MLPerf is a time-to-quality benchmark: batch-size choices change epochs-to-target (e.g., 64 epochs at 4K vs >80 at 16K for ResNet-50, +30% compute), and benchmarks expose interconnect/CPU bottlenecks | v0.5-era results (ResNet-50 target was 74.9% top-1 then, later 75.9%); reference implementations; mixed-precision improvement quantified qualitatively in abstract | high |
| rajbhandari2020zero | 2020 | system paper | 400 NVIDIA V100 GPUs; GPT-2/BERT-style models up to 170B params | training throughput in TFLOPS/Petaflops; model size; scaling efficiency | ZeRO-100B trains 100B-parameter models on 400 V100 GPUs at ~38 TFLOPS/GPU (15+ Petaflops aggregate, >30% of peak) with super-linear speedup — an 8x increase in model size and 10x increase in performance over state-of-the-art | ZeRO-100B evaluates only a subset of optimizations (Pos+g + ZeRO-R, not full Pos+g+p); the 1-trillion-parameter capability is analytical extrapolation, not a measured run | high |
| chen2016sublinearmemory | 2016 | system paper | 1,000-layer deep residual network on ImageNet; general CNNs/RNNs | training memory footprint (activation/gradient storage) | Algorithm reduces activation memory to O(sqrt(n)) for an n-layer network at the cost of one extra forward pass per mini-batch; a 1,000-layer ResNet drops from 48G to 7G memory on ImageNet | Trade-off adds compute (roughly double the forward computation); O(log n) memory variant costs O(n log n) extra forward compute; pre-GPU-memory-era hardware context | high |
| shoeybi2019megatron | 2019 | system paper | GPT-2-like 8.3B-param transformer and BERT up to 3.9B, on 512 NVIDIA V100 GPUs | sustained training throughput (PetaFLOPs); scaling efficiency | Simple intra-layer (tensor) model parallelism trains an 8.3B-parameter transformer on 512 GPUs at 15.1 sustained PetaFLOPs with 76% scaling efficiency vs single-GPU baseline | Intra-layer MP only (no pipeline); scaling shown to 512 GPUs; weak-scaling setup keeps ~1B params/GPU; requires NVLink-class intra-node bandwidth for efficiency | high |
| narayanan2021megatronsc21 | 2021 | system paper | GPT models from 1B to 1T parameters, up to 3072 NVIDIA A100 GPUs (384 DGX A100 nodes) | end-to-end training throughput (TFLOPS/GPU, aggregate petaFLOP/s, % of theoretical peak); estimated training time | Composing pipeline+tensor+data parallelism (PTD-P) with a new interleaved schedule trains a 1-trillion-parameter GPT at 163 TFLOPS/GPU end-to-end = 502 petaFLOP/s on 3072 A100s (52% of peak), with close-to-linear scaling | Training-time estimates (34/84 days) are projections, not wall-clock runs; requires high-bandwidth interconnects (892 GB/s PP, 13 TB/s DP effective bisection); parallelism choice is heuristic, not auto-searched | high |
| huang2019gpipe | 2018 | system paper | AmoebaNet-D 557M params (ImageNet); 6B-param 128-layer multilingual Transformer; Cloud TPUv2/v3 | training throughput speedup vs device count; top-1 accuracy; BLEU | GPipe's batch-splitting pipelining gives almost linear speedup with device count (3.5x for Transformer-48 across 4x accelerators) and trains a 557M-param AmoebaNet to 84.4% top-1 on ImageNet | Bubble overhead non-negligible when micro-batches < 4x partitions; AmoebaNet gets sub-linear speedup due to imbalanced computation; requires activation recomputation; limited to sequential-layer models | high |
| narayanan2019pipedream | 2019 | system paper | general DNN training workloads on multi-accelerator systems (abstract; no hardware specified) | training throughput; pipeline stalls; communication volume | PipeDream adds inter-batch pipelining on top of intra-batch parallelism, overlapping computation with communication and reducing communication, using parameter versioning for numerically correct gradients with minimal pipeline stalls | Abstract-only: no quantitative results available; claims are qualitative | moderate |
| li2021terapipe | 2021 | system paper | GPT-3 models up to 175B params; AWS cluster with 48 p3.16xlarge instances (384 V100 GPUs) | synchronous model-parallel training throughput (speedup) | Token-level pipeline parallelism (within a single training sequence, exploiting autoregressive structure) speeds up 175B-param GPT-3 training by 5.0x vs state-of-the-art model-parallel methods | Requires autoregressive Transformers with long sequences; optimal schedule computed per model+cluster by dynamic programming; V100-era hardware | high |
| smith2022mtnlg | 2022 | system paper | Megatron-Turing NLG 530B (530B params); up to 420 DGX A100 nodes (3,360 A100 80GB GPUs) on Selene; 270B training tokens | iteration time (s); per-GPU TFLOPS; model quality | MT-NLG 530B trains at 126-113 teraFLOP/s per GPU (60.1-44.4 s/iteration, batch 1920) scaling from 280 to 420 DGX A100 servers, i.e., ~40%-36% of the 312 TFLOPS A100 peak | Single flagship run without controlled ablations; per-GPU efficiency declines with scale (126 to 113 TFLOPS/GPU); results tied to Selene's fat-tree HDR InfiniBand fabric | high |
| bennun2019demystifying | 2019 | survey | n/a (general DNN architectures and parallelization approaches) | n/a (taxonomy/qualitative) | Survey modeling the concurrency types in DNNs from single operator to distributed training, covering async stochastic optimization, system architectures, and communication schemes, and extrapolating future parallelism directions | Abstract-only; no quantitative measurements; survey scope limits claim specificity | moderate |
| lepikhin2020gshard | 2020 | system paper | multilingual NMT Transformer up to 600B params (MoE, top-2 gating); 2048 TPU v3 cores | training cost (TPU v3 core-years, wall-clock days); translation quality (delta BLEU) | A 600B-parameter sparsely-gated MoE model trains on 2048 TPU v3 cores in 4 days (22 core-years) with better quality than 100 bilingual dense baselines (29 core-years) and far cheaper than a dense 2.3B model (235.5 core-years) | Single workload (NMT, proprietary corpus); MoE top-2 gating with auxiliary load-balancing loss; quality vs dense models compared on translation only | high |
| fedus2022switch | 2021 | system paper | T5-family Switch Transformers up to 1.6T params (Switch-C, 2048 experts); C4 corpus; TPUv3 | pre-training speedup (steps/step-time to quality); sample efficiency | Simplifying MoE routing to one expert per token yields up to 7x pre-training speedup over FLOP-matched T5-Base and 4x over T5-XXL at trillion-parameter scale, with a 1.6T-param model showing no training instability | TPU-centric evaluation (expert parallelism via all-to-all); some large sparse variants (Switch-XXL) show instability and need selective precision/smaller init; quality measured on C4 pre-training + downstream NLP | high |
| chen2025moefusion | 2025 | system paper | MoE models in distributed training with inter-GPU shared memory (abstract; no hardware specified) | communication overhead; expert load imbalance; computational efficiency | Proposes using inter-GPU shared memory for seamless MoE communication-computation fusion; argues existing coarse-grained overlap reduces communication but impairs computational efficiency, and load-balancing fixes often compromise model quality | Abstract-only; no quantitative results or evaluation details | moderate |
| deepseek2024deepseekv3 | 2024 | system paper | DeepSeek-V3 MoE (671B total / 37B activated per token); 2048 NVIDIA H800 GPUs; 14.8T pre-training tokens | GPU hours, USD cost, wall-clock days | Full training of DeepSeek-V3 costs 2.788M H800 GPU hours (~$5.576M at $2/GPU-hr), with pre-training (14.8T tokens) under two months on 2048 H800s at 180K GPU hours per trillion tokens (3.7 days) | Cost excludes ablations/research; H800 cluster (reduced NVLink) makes results hardware-specific; relies on heavy co-design (FP8, MLA, DualPipe, auxiliary-loss-free balancing) | high |
| mikami2019networkopt | 2019 | system paper | AlexNet and ResNet-50 on ImageNet-1K; 512 GPUs; 56 Gbps-era cluster networking | end-to-end training time to target accuracy; speedup ratio | GradientFlow communication backend (ring allreduce + mixed precision + overlap + lazy allreduce + coarse-grained sparse communication) trains 95-epoch ImageNet/AlexNet in 1.5 minutes on 512 GPUs (410.2x speedup) and 90-epoch ResNet-50 in 7.3 minutes (434.1x) | Internal system/cluster specifics (56 Gbps NICs); sparse/lazy allreduce trades gradient exactness; speedups vs per-GPU baselines on older hardware; accuracy thresholds not detailed in abstract | high |
| li2021iscaoverlap | 2021 | system paper | DL accelerators (GPU/TPU-class) with fast interconnects; ACE collective-communication accelerator engine | memory bandwidth demand; effective network bandwidth utilization; training speedup | ACE, a collective-communication accelerator at the accelerator endpoint, frees compute/memory for DL work: it reduces the memory bandwidth required to drive the same network bandwidth by 3.5x on average and raises effective network bandwidth utilization by 1.44x (up to 2.67x), yielding average training speedups around 1.41x (up to 1.51x) | Abstract truncated in source pack (third metric cut off mid-number); results from real-system measurements + detailed modeling (simulation); requires new endpoint hardware; source txt is an unparseable raw PDF so extraction is abstract-only | moderate |
| ghosh2022sc22overlap | 2022 | system paper | large DL models on GPU/TPU accelerator clusters; intra-layer model parallelism (abstract) | communication time share of execution; computational efficiency | Intra-layer model-parallel data communication can be a significant proportion of overall execution time; proposes overlapping communication with dependent computation via decomposition to recover computational efficiency | Abstract-only; no quantitative results | moderate |
| shen2024co2 | 2024 | system paper | TransNormer-LLM 7B, GPT-2, ResNet-50/ViT; up to 128 A100 GPUs (16 DGX nodes); RoCE RDMA ~800Gbps and TCP/IP ~80Gbps inter-node | throughput (words/sec); scalability ratio (16 to 128 GPUs); convergence | CO2 (local updating + asynchronous communication) achieves full communication-computation overlap and ~100% scaling: scalability ratio 1.08 vs AdamW's 0.93 on RDMA (16 to 128 GPUs), reaching 307,557 words/sec at 128 GPUs, and beating AdamW even on ~80Gbps TCP/IP | Local updating changes optimizer semantics (staleness controlled by tau and penalties); convergence bound relies on bounded-staleness assumptions; throughput measured over iterations 100-200 | high |
| li2024fluxoverlap | 2024 | system paper | Tensor-parallel LLM training (GPT-3-175B-derived GEMM shapes) and prefill/decoding inference on A100/H800 clusters (128 GPUs training, 8 GPUs inference) | communication-overlap efficiency and end-to-end speedup | FLUX fuses over-decomposed communication+compute kernels to overlap up to 96% of tensor-parallel communication, yielding up to 1.24x training speedup over Megatron-LM and up to 1.66x prefill / 1.30x decoding speedup over vLLM | Speedups measured on a limited set of GEMM shapes/sequence lengths; several H800 small-shape decoding cases show no gain or slight slowdown (0.95x); when TP comm is a small time share, model-level gains shrink to ~1.04x-1.05x | high |
| ispass2025overlapchar | 2025 | benchmark study | GPT and LLaMA models on NVIDIA H100/A100 and AMD MI250/MI210 GPUs with varied precision, specialized cores, and power capping | compute slowdown / speedup from overlapping communication with compute, and power effects | Overlapping communication with compute causes an average 18.9% (up to 40.0%) compute slowdown versus ideal non-interfered execution, but is still on average 10.2% (max 26.6%) faster than sequential execution | Abstract-only; power-capping and specialized-datapath effects quantified only qualitatively in abstract; results averaged across heterogeneous GPU families | moderate |
| yuan2024centauri | 2024 | system paper | Hybrid-parallel (data/tensor/pipeline) LLM training with multiple communication collectives | scheduling of communication-computation overlap | Centauri partitions communication to enable efficient overlap scheduling in heterogeneous training environments, overcoming limits of fine-grained kernel fusion and limited operation scheduling | Abstract-only and truncated; no quantitative speedups reported in the abstract | low |
| li20221bitlamb | 2022 | system paper | BERT/GPT-3-scale models trained on hundreds of GPUs over limited-bandwidth TCP networks | wall-clock training time under compressed communication + large-batch optimization | 1-bit LAMB combines large-batch optimization (LAMB) with 1-bit communication compression, adding support for adaptive layerwise learning rates under compression plus three system implementations to achieve real wall-clock speedup | Abstract-only; no speedup magnitudes or convergence numbers in the abstract | moderate |
| mohan2021datastalls | 2021 | empirical study | Nine CV/audio DNN models, three tasks, four datasets on Microsoft production cluster servers varying memory, CPU threads, storage device, GPU generation | data stall time (waiting for data fetch/pre-processing) as share of DNN training time | In many cases DNN training time is dominated by data stall time; the paper builds DS-Analyzer to measure stalls via a differential technique and to run predictive what-if analysis | Full-text file is a raw PDF (no extractable text), so quantitative figures could not be verified and are omitted; findings limited to abstract-level claims | moderate |
| ma2025tensorsocket | 2025 | system paper | Collocated image-classification training workloads (MobileNet, RegNetX variants) on NVIDIA GPUs (incl. A100) sharing one data loader | training throughput, CPU/GPU utilization, cloud instance cost | TensorSocket shares one data loader across simultaneous training processes over GPU-GPU interconnects, increasing training throughput by up to 100% and cutting cloud CPU-side cost by ~50% | Evaluation focuses on collocated same-machine scenarios; throughput-per-process drops in some online-training collocation setups; hardware- and pipeline-agnostic claims based on specific workload mixes | high |
| li2019hipcdataload | 2019 | system paper | Large-scale distributed DNN training (256 nodes, 1,024 learners) with CPU-side data loading | data loading time and I/O communication volume | Identifies data loading as a scalability limiter and, via an analytical model plus locality-aware software-cache design, achieves more than 30x speedup in data loading on 256 nodes / 1,024 learners | Full-text file is a raw PDF (no extractable text); the 30x figure and other numbers come from the abstract only; numbers are data-loading speedup, not end-to-end training speedup | moderate |
| mlcommons2022mlperfstorage | 2022 | benchmark study | MLPerf Storage benchmark workloads (training and inference I/O patterns) | I/O and storage performance characterization of ML workloads | Introduces the MLPerf Storage benchmark to characterize how data ingest, storage, and serving affect end-to-end ML performance, motivated by the growing infeasibility of keeping training datasets in main memory | Abstract-only; no quantitative results in the abstract | low |
| dao2022flashattention | 2022 | system paper | Transformer training/inference (BERT-large, GPT-2, Long-Range Arena) on A100 GPUs | attention kernel runtime and end-to-end training wall-clock time | IO-aware tiling of attention reduces HBM<->SRAM traffic, giving linear memory use and 3x GPT-2 / 2.4x Long-Range-Arena speedups, and a 15% end-to-end BERT-large speedup vs the MLPerf 1.1 training speed record | Numbers depend on sequence length and GPU generation; speedups are vs PyTorch/standard baselines; approximate variants (block-sparse) trade exactness | high |
| dao2023flashattention2 | 2023 | system paper | Attention kernels for Transformer training/inference (seq lengths 2k-8k and longer) on A100 80GB SXM4 | attention kernel runtime, achieved FLOPs/s, end-to-end training speedup | FlashAttention-2 reworks parallelism/work partitioning to reach ~2x over FlashAttention (up to 225 TFLOPs/s per A100, 50-73% of peak), with up to 10x over standard attention and 10-20x memory savings | Speedups vary with sequence length/head count; end-to-end gains (up to 1.3x over FA, 2.8x over no-FA) smaller than kernel-level gains; further H100 gains (1.5x-2x) left as future work | high |
| naumov2019dlrm | 2019 | system paper | DLRM recommendation model on Criteo Ad Kaggle (~45M samples) and Terabyte datasets; multi-GPU training | embedding memory footprint and distributed-training communication pattern | DLRM's embedding tables hold the majority of parameters (multiple GBs per table), making data parallelism prohibitive and forcing model-parallel embedding sharding with personalized all-to-all communication | Design/analysis paper; no end-to-end training-time speedups vs baselines reported in the extracted text | moderate |
| sridhar2024dlrmtt | 2024 | system paper | Distributed DLRM training with Tensor-Train (TT)-decomposed embedding tables (EcoRec library on PyTorch/CUDA) | embedding memory footprint, TT-op compute time, and communication efficiency in distributed training | EcoRec synergizes TT decomposition with distributed training via redundancy-free TT computation, sorted-index micro-batching, and pipelined embedding-layer training to cut compute and communication overhead | Abstract-only (truncated); no quantitative speedups or memory numbers in the abstract | low |
| hu2022lora | 2021 | empirical study | Fine-tuning of RoBERTa, DeBERTa, GPT-2, and GPT-3 175B with frozen weights + rank-decomposition adapters | trainable parameter count, GPU VRAM, training throughput, task accuracy | LoRA cuts trainable parameters 10,000x and GPU memory 3x for GPT-3 175B fine-tuning, gives a 25% training speedup, and matches or beats full fine-tuning quality | Speedup/memory numbers are specific to GPT-3 175B with r=4 adapting query/value projections; quality parity shown on a limited task set; LoRA applied to attention weights only in main experiments | high |
| parker2024breakingmlperf | 2024 | case study | MLPerf Training v1.1/v2.0 BERT closed benchmark on 1,024 NVIDIA A100 GPUs (100 GB/s network) | time to MLPerf BERT target accuracy (72.0%) | Combining local presorting (dataset stratification), bucket-wise gradient clipping before allreduce, and re-optimized ADAM yields 25.1 (22.3) seconds on 1,024 A100s — the fastest MLPerf BERT training, 1.33x (1.13x) and 1.57x faster than the other top submissions to v1.1 (v2.0) | Single benchmark/model (BERT) on one cluster configuration; results depend on 100 GB/s network without SHARP; comparison baselines are other MLPerf submissions, not all possible systems | high |

## References

1. Verma, Snehil et al. (2020). *Demystifying the MLPerf Training Benchmark Suite*. IEEE ISPASS 2020. Full text read. Defines the MLPerf Training benchmark: time-to-quality metric, closed-division equivalence rules, and epochs-vs-batch-size data (R50: ~64 epochs at 4K vs >80 at 16K). [doi:10.1109/ispass48437.2020.00013](https://doi.org/10.1109/ispass48437.2020.00013)
2. Kim, Yongdeok et al. (2024). *Breaking MLPerf Training: A Case Study on Optimizing BERT*. arXiv (Cornell University). Full text read. Case study of the fastest MLPerf BERT closed submission (25.1 s on 1024 A100s): stratification presorting, bucket-wise clip-before-allreduce, Adam re-tuning. [doi:10.48550/arxiv.2402.02447](https://doi.org/10.48550/arxiv.2402.02447)
3. Goyal, Priya et al. (2017). *Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour*. arXiv (Cornell University). Full text read. Foundational large-batch result: linear LR scaling + gradual warmup train ImageNet ResNet-50 at 8k batch in 1 hour on 256 GPUs without accuracy loss. [doi:10.48550/arxiv.1706.02677](https://doi.org/10.48550/arxiv.1706.02677)
4. Keskar, Nitish Shirish et al. (2016). *On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima*. arXiv (Cornell University). Full text read. Early evidence that large-batch training generalizes worse (gap up to 5%), attributed to convergence to sharp minima; the sharp-minima explanation was later contested. [doi:10.48550/arxiv.1609.04836](https://doi.org/10.48550/arxiv.1609.04836)
5. You, Yang et al. (2019). *Large Batch Optimization for Deep Learning: Training BERT in 76 minutes*. arXiv (Cornell University). Full text read. LAMB layerwise-adaptive optimizer enables BERT batch 32,868, cutting pretraining from 3 days to 76 minutes on a TPUv3 Pod. [doi:10.48550/arxiv.1904.00962](https://doi.org/10.48550/arxiv.1904.00962)
6. Nado, Zachary et al. (2021). *A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes*. arXiv (Cornell University). Full text read. Contests LAMB/LARS specialness under MLPerf rules: tuned Nesterov matches LARS on ResNet-50 and Adam beats LAMB on BERT at batch 65,536. [doi:10.48550/arxiv.2102.06356](https://doi.org/10.48550/arxiv.2102.06356)
7. Merrill, William et al. (2025). *Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training*. Advances in Neural Information Processing Systems 38. Full text read. Measures critical batch size over training; batch-size warmup trains OLMo 1B with 43% fewer gradient steps. [doi:10.52202/085713-3898](https://doi.org/10.52202/085713-3898)
8. Hoffmann, Jordan et al. (2022). *Training Compute-Optimal Large Language Models*. arXiv (Cornell University). Full text read. Compute-optimal scaling: model size and tokens should scale equally; large models are undertrained — context for fixed-model convergence targets. [doi:10.48550/arxiv.2203.15556](https://doi.org/10.48550/arxiv.2203.15556)
9. Micikevicius, Paulius et al. (2017). *Mixed Precision Training*. arXiv (Cornell University). Full text read. The mixed-precision recipe (FP16 + FP32 master weights + loss scaling + FP32 accumulate) that matches FP32 accuracy without retuning. [doi:10.48550/arxiv.1710.03740](https://doi.org/10.48550/arxiv.1710.03740)
10. Kalamkar, Dhiraj et al. (2019). *A Study of BFLOAT16 for Deep Learning Training*. arXiv (Cornell University). Full text read. BF16 training matches FP32 results with no hyperparameter changes across five task families. [doi:10.48550/arxiv.1905.12322](https://doi.org/10.48550/arxiv.1905.12322)
11. Markidis, Stefano et al. (2018). *NVIDIA Tensor Core Programmability, Performance &amp; Precision*. IEEE IPDPSW 2018. Full text read. Documents V100 Tensor Core programmability and the ~7x mixed-precision throughput advantage that makes low-precision training the default. [doi:10.1109/ipdpsw.2018.00091](https://doi.org/10.1109/ipdpsw.2018.00091)
12. Micikevicius, Paulius et al. (2022). *FP8 Formats for Deep Learning*. arXiv (Cornell University). Full text read. Defines FP8 E4M3/E5M2 formats; FP8 training matches FP16/BF16 quality up to 175B-parameter models with unchanged hyperparameters. [doi:10.48550/arxiv.2209.05433](https://doi.org/10.48550/arxiv.2209.05433)
13. Peng, Houwen et al. (2023). *FP8-LM: Training FP8 Large Language Models*. arXiv (Cornell University). Full text read. FP8 automatic mixed precision trains GPT-175B 75% faster than BF16 Megatron-LM with 39% lower memory. [doi:10.48550/arxiv.2310.18313](https://doi.org/10.48550/arxiv.2310.18313)
14. Wang, Ruizhe et al. (2025). *Optimizing Large Language Model Training Using FP4 Quantization*. arXiv (Cornell University). Full text read. First FP4 LLM training framework; accuracy comparable to BF16/FP8 at up to 13B parameters / 100B tokens. [doi:10.48550/arxiv.2501.17116](https://doi.org/10.48550/arxiv.2501.17116)
15. Croci, Matteo et al. (2022). *Stochastic rounding: implementation, error analysis and applications*. Royal Society Open Science. Abstract only. Error analysis of stochastic rounding (the technique the closed division explicitly allows): sqrt(n)*u error bounds and immunity to stagnation. [doi:10.1098/rsos.211631](https://doi.org/10.1098/rsos.211631)
16. Zhang, Sai Qian et al. (2022). *FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Rounding*. IEEE HPCA 2022. Abstract only. Variable-precision block floating point with stochastic rounding gives 2-6x training speedups at similar accuracy on a single-chip platform. [doi:10.1109/hpca53966.2022.00067](https://doi.org/10.1109/hpca53966.2022.00067)
17. Hao, Zhiwei et al. (2026). *Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities*. IEEE Transactions on Pattern Analysis and Machine Intelligence. Full text read. Survey of low-precision LLM training across fixed-point, floating-point, and custom formats; FP8/FP4 support in Hopper/Blackwell. [doi:10.1109/tpami.2026.3718968](https://doi.org/10.1109/tpami.2026.3718968)
18. Rajbhandari, Samyam et al. (2020). *ZeRO: Memory optimizations Toward Training Trillion Parameter Models*. ACM/IEEE SC20. Full text read. ZeRO partitions optimizer state, gradients, and parameters across data-parallel ranks, training 100B-parameter models on 400 V100s at 15+ Petaflops. [doi:10.1109/sc41405.2020.00024](https://doi.org/10.1109/sc41405.2020.00024)
19. Chen, Tianqi et al. (2016). *Training Deep Nets with Sublinear Memory Cost*. arXiv (Cornell University). Full text read. Activation recomputation reduces memory to O(sqrt(n)) at one extra forward pass — the recomputation the rules count as non-model flops. [doi:10.48550/arxiv.1604.06174](https://doi.org/10.48550/arxiv.1604.06174)
20. Shoeybi, Mohammad et al. (2019). *Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism*. arXiv (Cornell University). Full text read. Intra-layer tensor parallelism trains an 8.3B-parameter transformer on 512 GPUs at 15.1 Petaflops sustained (76% scaling efficiency). [doi:10.48550/arxiv.1909.08053](https://doi.org/10.48550/arxiv.1909.08053)
21. Narayanan, Deepak et al. (2021). *Efficient large-scale language model training on GPU clusters using megatron-LM*. ACM/IEEE International Conference for High Performance Computing (SC21). Full text read. PTD-P with interleaved pipeline schedule trains a 1T-parameter GPT at 502 petaFLOP/s on 3072 A100s (52% of peak). [doi:10.1145/3458817.3476209](https://doi.org/10.1145/3458817.3476209)
22. Huang, Yanping et al. (2018). *GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism*. arXiv (Cornell University). Full text read. Batch-splitting pipeline parallelism with ~linear speedup (3.5x for Transformer across 4x accelerators). [doi:10.48550/arxiv.1811.06965](https://doi.org/10.48550/arxiv.1811.06965)
23. Narayanan, Deepak et al. (2019). *PipeDream*. ACM SOSP 2019. Abstract only. Inter-batch pipelining with parameter versioning to keep gradients numerically correct with minimal stalls. [doi:10.1145/3341301.3359646](https://doi.org/10.1145/3341301.3359646)
24. Li, Zhuohan et al. (2021). *TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models*. arXiv (Cornell University). Full text read. Token-level pipeline parallelism exploiting autoregression speeds GPT-3 175B training 5.0x on 48 AWS instances. [doi:10.48550/arxiv.2102.07988](https://doi.org/10.48550/arxiv.2102.07988)
25. Smith, Shaden et al. (2022). *Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model*. arXiv (Cornell University). Full text read. Megatron-Turing NLG 530B: 126 down to 113 TFLOPS/GPU scaling 280 to 420 DGX A100 servers — quantifies the efficiency tax of scale. [doi:10.48550/arxiv.2201.11990](https://doi.org/10.48550/arxiv.2201.11990)
26. Ben‐Nun, Tal & Hoefler, Torsten (2019). *Demystifying Parallel and Distributed Deep Learning*. ACM Computing Surveys. Abstract only. Survey/taxonomy of parallelism in deep learning from operator level to distributed training. [doi:10.1145/3320060](https://doi.org/10.1145/3320060)
27. Lepikhin, Dmitry et al. (2020). *GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding*. arXiv (Cornell University). Full text read. GShard: 600B-parameter MoE trained on 2048 TPUv3s in 4 days (22 core-years) vs 235.5 core-years for a dense 2.3B baseline. [doi:10.48550/arxiv.2006.16668](https://doi.org/10.48550/arxiv.2006.16668)
28. Fedus, William et al. (2021). *Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity*. arXiv (Cornell University). Full text read. Single-expert routing gives up to 7x pretraining speedup at fixed FLOPs; 1.6T-parameter Switch-C trains stably. [doi:10.48550/arxiv.2101.03961](https://doi.org/10.48550/arxiv.2101.03961)
29. Wang, H. et al. (2025). *Harnessing Inter-GPU Shared Memory for Seamless MoE Communication-Computation Fusion*. ACM SIGPLAN PPoPP 2025. Abstract only. Inter-GPU shared memory for MoE communication-computation fusion; argues coarse overlap impairs compute efficiency. [doi:10.1145/3710848.3710868](https://doi.org/10.1145/3710848.3710868)
30. DeepSeek-AI et al. (2024). *DeepSeek-V3 Technical Report*. arXiv (Cornell University). Full text read. DeepSeek-V3: 671B/37B-active MoE trained for 2.788M H800 GPU hours ($5.576M) using FP8, DualPipe overlap, aux-loss-free balancing — the v6.0 closed benchmark model. [doi:10.48550/arxiv.2412.19437](https://doi.org/10.48550/arxiv.2412.19437)
31. Sun, Peng et al. (2019). *Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes*. arXiv (Cornell University). Full text read. GradientFlow: ring allreduce + overlap + mixed precision trains ImageNet/AlexNet 95 epochs in 1.5 min (410.2x) and ResNet-50 in 7.3 min (434.1x) on 512 GPUs. [doi:10.48550/arxiv.1902.06855](https://doi.org/10.48550/arxiv.1902.06855)
32. Rashidi, Saeed et al. (2021). *Enabling Compute-Communication Overlap in Distributed Deep Learning Training Platforms*. ACM/IEEE International Symposium on Computer Architecture (ISCA 2021). Full text read. ACE collective-communication accelerator cuts memory bandwidth demand 3.5x and raises effective network bandwidth utilization 1.44x (up to 2.67x). [doi:10.1109/isca52012.2021.00049](https://doi.org/10.1109/isca52012.2021.00049)
33. Wang, Shibo et al. (2022). *Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models*. ACM/IEEE SC22. Abstract only. Decomposition-based overlap of dependent communication in intra-layer model parallelism. [doi:10.1145/3567955.3567959](https://doi.org/10.1145/3567955.3567959)
34. Sun, Weigao et al. (2024). *CO2: Efficient Distributed Training with Full Communication-Computation Overlap*. arXiv (Cornell University). Full text read. Local updating + asynchronous communication achieves full overlap: scalability ratio 1.08 vs AdamW 0.93 from 16 to 128 GPUs. [doi:10.48550/arxiv.2401.16265](https://doi.org/10.48550/arxiv.2401.16265)
35. Chang, Li‐Wen et al. (2024). *FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion*. arXiv (Cornell University). Full text read. Kernel fusion of over-decomposed communication+compute overlaps up to 96% of tensor-parallel communication; up to 1.24x training speedup. [doi:10.48550/arxiv.2406.06858](https://doi.org/10.48550/arxiv.2406.06858)
36. Lee, Seonho et al. (2025). *Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implications*. IEEE ISPASS 2025. Abstract only. Characterizes overlap costs: 18.9% average compute slowdown (up to 40%) but still 10.2% faster than sequential; power capping exacerbates. [doi:10.1109/ispass64960.2025.00041](https://doi.org/10.1109/ispass64960.2025.00041)
37. Chen, Chang et al. (2024). *Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning*. ACM ASPLOS 2024. Abstract only. Communication partitioning enables overlap scheduling in hybrid-parallel training where fusion or op-scheduling alone falls short. [doi:10.1145/3620666.3651379](https://doi.org/10.1145/3620666.3651379)
38. Li, Conglong et al. (2022). *1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB’s Convergence Speed*. IEEE HiPC 2022. Abstract only. Combines LAMB large-batch training with 1-bit gradient compression for low-bandwidth clusters; compression itself is likely outside closed-division equivalence. [doi:10.1109/hipc56025.2022.00044](https://doi.org/10.1109/hipc56025.2022.00044)
39. Mohan, Jayashree et al. (2021). *Analyzing and mitigating data stalls in DNN training*. Proceedings of the VLDB Endowment. Full text read. Production study at Microsoft: DNN training time is often dominated by data-stall time (fetch + preprocessing). [doi:10.14778/3446095.3446100](https://doi.org/10.14778/3446095.3446100)
40. Robroek, Ties et al. (2025). *TensorSocket: Shared Data Loading for Deep Learning Training*. Proceedings of the ACM on Management of Data. Full text read. Sharing one data loader across collocated training processes raises throughput up to 100% and cuts cloud CPU cost ~50%. [doi:10.1145/3749185](https://doi.org/10.1145/3749185)
41. Yang, Chih-Chieh & Cong, Guojing (2019). *Accelerating Data Loading in Deep Neural Network Training*. IEEE HiPC 2019. Full text read. Locality-aware data loading with software caches gives >30x data-loading speedup on 256 nodes / 1,024 learners. [doi:10.1109/hipc.2019.00037](https://doi.org/10.1109/hipc.2019.00037)
42. Balmau, Oana (2022). *Characterizing I/O in Machine Learning with MLPerf Storage*. ACM SIGMOD Record. Abstract only. Argues storage/preprocessing is the neglected axis of ML performance as datasets outgrow main memory; introduces MLPerf Storage. [doi:10.1145/3572751.3572765](https://doi.org/10.1145/3572751.3572765)
43. Dao, Tri et al. (2022). *FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness*. Advances in Neural Information Processing Systems 35. Full text read. IO-aware attention tiling: 15% end-to-end BERT-large speedup vs the MLPerf 1.1 training record, 3x GPT-2, linear memory. [doi:10.52202/068431-1189](https://doi.org/10.52202/068431-1189)
44. Dao, Tri (2023). *FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning*. arXiv (Cornell University). Full text read. Better parallelism/work partitioning: ~2x over FlashAttention, up to 225 TFLOPS/s per A100 (50-73% of peak). [doi:10.48550/arxiv.2307.08691](https://doi.org/10.48550/arxiv.2307.08691)
45. Naumov, Maxim et al. (2019). *Deep Learning Recommendation Model for*. arXiv (Cornell University). Full text read. Defines DLRM; embedding tables dominate memory and force model-parallel sharding with personalized all-to-all. [doi:10.48550/arxiv.1906.00091](https://doi.org/10.48550/arxiv.1906.00091)
46. Wang, Weihu et al. (2024). *Accelerating Distributed DLRM Training with Optimized TT Decomposition and Micro-Batching*. ACM/IEEE SC24. Abstract only. TT-decomposed embeddings + sorted-index micro-batching + pipelined embedding training to cut DLRM compute and communication. [doi:10.1109/sc41406.2024.00055](https://doi.org/10.1109/sc41406.2024.00055)
47. Hu, J. Edward et al. (2021). *LoRA: Low-Rank Adaptation of Large Language Models*. arXiv (Cornell University). Full text read. LoRA fine-tuning: 10,000x fewer trainable parameters, 3x less memory (1.2TB to 350GB), 25% training speedup on GPT-3 175B. [doi:10.48550/arxiv.2106.09685](https://doi.org/10.48550/arxiv.2106.09685)