Optimizing MLPerf Training closed-division submissions: what the literature shows

Why this question

MLPerf Training is the only widely recognized benchmark where a closed-division submission must reproduce a reference model’s exact graph, initialization, and quality target while still racing the clock. Submitters (NVIDIA, Google, and a handful of others) publish results but not playbooks; the rules document defines what is legal but not what is fast. Anyone preparing a submission — including the author of this review — needs the transferable evidence: which published techniques are (a) effective at reducing wall-clock time to quality and (b) still inside the closed division’s equivalence rules.

What turns on the answer is real money and engineering time: a v6.0-scale submission (Llama 3.1 405B or DeepSeek-V3 671B pretraining) costs millions of GPU-hours, and a wrong assumption about what the rules permit — say, adopting a communication-compression scheme that changes the optimizer’s math — can invalidate a submission at review time. The review is therefore as much a compliance map as a performance survey.

Scope and methods

Question. Which methods documented in the literature reduce time-to-quality for MLPerf Training closed-division submissions, given fixed model, optimizer, and most hyperparameters?

Population. MLPerf Training closed-division benchmarks across versions (ResNet-50, BERT, GPT-3, Llama 3.1 8B/405B, GPT-OSS-20B, DeepSeek-V3-671B, DLRMv2, Llama2-70B-LoRA, FLUX.1). Concepts. Large-batch training and learning-rate schedules; mixed and reduced precision (fp16/bf16/tf32/fp8/fp4, stochastic rounding); memory optimization and parallelism (data/tensor/pipeline/expert); communication optimization and overlap; data loading; kernel-level optimization; benchmark-specific engineering. Outcomes. Wall-clock time to quality target, epochs-to-convergence, throughput, speedup ratios. Inclusion. Peer-reviewed systems/ML papers, arXiv preprints, and benchmark-definition documents from 2015–2026, English. Exclusion. Inference benchmarks, MLPerf HPC/Storage/Inference variants (one storage characterization paper kept for the I/O argument), open-division model changes, gradient-compression as a recommended lever (flagged where relevant), hardware microarchitecture papers without training results.

Search and screening. 20 layered queries (core-concept + synonym + filter blocks) across OpenAlex, Crossref, arXiv, and Semantic Scholar on 2026-08-07 (Semantic Scholar was rate-limited and returned nothing usable; OpenAlex’s budget exhausted mid-run and was completed from a second IP). Raw retrieval 2,594 records; 2,350 unique after DOI/title dedup; 2,291 after topical exclusion of off-topic hits. A targeted title-resolution pass over 50 known-candidate papers (Crossref bibliographic + OpenAlex title.search) recovered the canonical literature that keyword search missed; three of my own remembered arXiv IDs resolved to different papers and were discarded after abs-page verification. Full-text screening and evidence extraction included 47 sources (30 read in full text, 17 abstract-only). Every DOI was verified against Crossref or DataCite this session; no source is retracted. Numbers cited here were spot-checked against full texts by grep; unverifiable figures were dropped.

The landscape

The literature splits into two largely disconnected bodies. The first is the systems literature (SC, ISCA, ASPLOS, MLSys, VLDB): how to make a fixed training workload run fast at scale — parallelism, communication, memory, I/O. The second is the optimization/ML literature (ICLR, NeurIPS, arXiv): how many steps a given model needs to reach quality, and which hyperparameters control that. MLPerf-specific work is tiny: the benchmark definition paper ¹, one published case study of a record BERT submission ², one storage benchmark ⁴², and a handful of cost-prediction/energy papers. Almost everything a submitter needs must be assembled from adjacent fields.

The corpus is dominated by three institutions — NVIDIA (mixed precision, Megatron, FlashAttention, network optimization), Microsoft (ZeRO, DeepSpeed, data stalls, 1-bit LAMB), and Google (GPipe, GShard, Switch, Chinchilla) — so “consensus” in places is one lab’s program replicated by others. The field is young: the oldest load-bearing result here is 2016, and the median is 2021–2022, meaning most evidence predates the current v6.0 benchmark suite (Llama 3.1 405B, DeepSeek-V3 671B, FLUX.1). One era dominates: the large-batch/optimizer debate (2016–2021) and the extreme-scale systems work (2020–2024) are mature; the fp4-era and v6-era submission analysis are open territory.

The rules frame everything that follows, so they are quoted here as the constraint set: the MLPerf Training Rules v6.0 require the same preprocessing, model graph, initialization, loss, and quality target as the reference; hyperparameters are fixed except a short tunable list; the optimizer implementation must match the reference; and only numerically safe precision-compensation techniques are pre-approved (scaling, orthogonal transforms like Hadamard, stochastic rounding). Everything else in this review is read against that boundary.

What the closed division actually fixes

The benchmark definition paper makes the central design explicit: MLPerf Training measures time to a quality target, not throughput, and the closed division enforces equivalence to the reference implementation precisely so that faster hardware and engineering — not clever model changes — are what win ¹. That framing has three consequences that shape every technique below.

First, convergence speed is capped, not rewarded. The rules require submissions to match Reference Convergence Points (RCPs): a one-sided t-test against reference convergence data allows only ~3.5–4% faster mean epochs-to-convergence (the rules’ worked example), and any submission converging faster is normalized to the RCP mean. The MLPerf paper documents the underlying phenomenon — epochs-to-target grows with batch size, e.g. ResNet-50 needing ~64 epochs at 4K batch versus more than 80 at 16K, a 30% compute increase ¹. So the score is effectively (RCP-mean epochs) × (time per epoch), and the first-order lever is the batch size you pick, because it fixes the epoch budget you are allowed.

Second, the clock is narrower than it looks. Model initialization (graph compile, checkpoint load) is untimed up to 30 minutes in closed division; only per-run preprocessing, training, and quality evaluation are on the clock; and for benchmarks with timestamped checkpoints, evaluation can run off-clock with the stop time rolled back to the first passing checkpoint. Nothing in the reviewed literature studies this explicitly, but the rules themselves note that coarse evaluation spacing cost v5.0-era Llama 405B submissions a 12–16% overhead when convergence landed just after an evaluation, which is why v6.0 evaluation frequencies were tightened.

Third, the optimizer is fixed but its implementation is a differentiator. The rules name the optimizer per benchmark (AdamW for the LLM benchmarks, Adagrad for DLRMv2) and require the implementation to match the approved list, with alternate implementations needing an equivalence demonstration. Within that, the case-study literature shows how much implementation quality matters: the record BERT submission re-tuned Adam (LR, betas, weight decay, and gradient-clip placement) inside the closed rules and beat the previous best closed submissions by 1.33× (v1.1) and 1.57× (v2.0) ².

Batch size: the highest-leverage tunable

The global batch size is the one hyperparameter the rules leave essentially unconstrained for most benchmarks, and it is the one with the most direct evidence of wall-clock effect. The canonical result is Goyal et al.: with linear learning-rate scaling and gradual warmup, ResNet-50 trains at an 8k batch in one hour on 256 GPUs with accuracy statistically indistinguishable from the small-batch baseline (23.74% vs 23.60% top-1 error) ³. The warmup detail matters — without it the 8k run degrades by over a point — and the same warmup-plus-scaling pattern recurs in every later large-batch recipe.

The early literature warned that large batches generalize worse: Keskar et al. measured gaps up to 5% and attributed them to convergence to sharp minima ⁴. That claim is contested (see below), and the practical resolution came from the optimizer line: LAMB’s layerwise-adaptive rates enabled BERT pretraining at batch 32,868, cutting time from 3 days to 76 minutes on a TPUv3 Pod ⁵, and a careful 2021 study under the actual MLPerf ResNet-50 rules showed that plain Nesterov momentum reaches the 75.9% target in 2,512 steps at batch 32,768 — matching the best published LARS result — while tuned Adam beats LAMB on BERT at batch 65,536 (F1 90.46 in 7,818 steps) ⁶. The generalizable finding is that the optimizer’s identity matters less than having one that tolerates the batch size, and that the “special large-batch optimizers” narrative overstated the gap.

For picking the batch size itself, the modern evidence is the critical-batch-size (CBS) literature: CBS starts near zero, rises quickly, then plateaus, and batch-size warmup — start small, grow with CBS — trained OLMo 1B to slightly better loss with 43% fewer gradient steps ⁷. The rules-compatible reading is awkward: closed-division runs must hold a single GBS for RCP matching, so the CBS result is a pointer toward which fixed GBS to choose (roughly the plateau value for the target loss), not a license to schedule batch growth mid-run — a mapping the literature has not tested. Chinchilla’s compute-optimality result (model size and tokens should scale equally) ⁸ is background context: the v6.0 LLM benchmarks are deliberately undertrained relative to compute-optimal, which is precisely why their RCPs allow hitting the target in a bounded token budget.

Precision: throughput that costs nothing

Reduced precision is the cleanest lever in the whole review because the rules pre-approve the formats and the evidence shows accuracy parity at unchanged hyperparameters. The foundation is the mixed-precision recipe — fp16 storage with fp32 master weights, loss scaling, fp32 accumulation — which matched fp32 accuracy without retuning across vision, speech, and language workloads ⁹, with bf16 achieving the same parity while keeping fp32’s dynamic range and thus requiring zero hyperparameter changes ¹⁰. The hardware side was established by the Tensor Core programmability work: mixed-precision tensor cores deliver roughly 7× single-precision throughput on V100-class hardware ¹¹, which is why every modern submission is precision-first.

The current frontier is fp8: the format definitions paper showed fp8 (E4M3/E5M2) training matches fp16/bf16 quality up to 175B parameters with unchanged hyperparameters ¹², and the FP8-LM system made it concrete — GPT-175B trains 75% faster than the bf16 Megatron-LM baseline with 39% less memory ¹³. DeepSeek-V3 validated fp8 at extreme scale in production: 671B parameters, 2.788M H800 GPU-hours total (~$5.6M at $2/GPU-hour), with fp8 mixed precision as a core co-design ³⁰. Below fp8, the first fp4 training framework reports accuracy comparable to bf16/fp8 at up to 13B parameters and 100B tokens ¹⁴ — notable because the v6.0 rules already pre-approve nvfp4/mxfp4. A 2026 survey consolidates the space: fixed-point, floating-point, and custom formats, with fp8/fp4 native paths on Hopper/Blackwell ¹⁷.

The numerically safe compensation techniques the rules explicitly allow are the subject of their own literature. Stochastic rounding — cited by name in the rules — has error bounds of order √n·u with high probability versus n·u worst case for round-to-nearest, and is immune to stagnation, the loss of tiny updates to large accumulators ¹⁵. Systems work shows the payoff: variable-precision block floating point with stochastic rounding delivers 2–6× training speedups at similar accuracy ¹⁶. The boundary to respect is the rules’ own: scaling and rounding are allowed; anything that changes the model’s real-valued function (e.g., altered normalization or gradient clipping) is not.

Memory and parallelism: fitting the model on the machine

For the large-model benchmarks (405B dense, 671B MoE), the first question is whether the model fits at all, and the answer is a 2020–2022 stack: ZeRO partitions optimizer state, gradients, and parameters across data-parallel ranks, training 100B-parameter models on 400 V100s at 38 TFLOPS/GPU with super-linear speedup ¹⁸; activation recomputation reduces activation memory to O(√n) at the cost of one extra forward pass — a 1,000-layer ResNet drops from 48 GB to 7 GB ¹⁹, and the rules’ utilization appendix explicitly counts recomputation flops as non-model flops, signaling it is expected. Tensor parallelism (Megatron-LM) carries models that data parallelism cannot: 8.3B parameters on 512 GPUs at 15.1 sustained Petaflops with 76% scaling efficiency ²⁰.

The composition playbook is Megatron’s SC21 paper: pipeline + tensor + data parallelism with an interleaved schedule trains a 1-trillion-parameter GPT at 502 petaFLOP/s on 3072 A100s (52% of theoretical peak, 163 TFLOPS/GPU end-to-end), with the interleaved schedule alone worth up to 10% and microbatch count up to 15% ²¹. The pipeline lineage that makes this possible: GPipe’s batch-splitting with near-linear speedup ²², PipeDream’s parameter versioning for numerically correct gradients ²³, and TeraPipe’s token-level pipelines for autoregressive models (5.0× on GPT-3 175B) ²⁴. The cost of scale is documented in the 530B run: per-GPU efficiency falls from 126 to 113 TFLOPS/GPU as the cluster grows from 280 to 420 DGX A100 servers ²⁵ — the same scaling tax the closed division inherits, since the reference implementations (NeMo, Megatron-Bridge) define the parallelism baseline a submission must beat with engineering, not architecture. The survey literature ²⁶ provides the organizing taxonomy for all of this.

For the MoE benchmarks (GPT-OSS-20B, DeepSeek-V3-671B), the sparse-axis evidence is strong: GShard trained a 600B-parameter MoE on 2048 TPUv3s in 4 days (22 core-years) versus 235.5 core-years for a dense 2.3B baseline ²⁷; Switch’s single-expert routing gave up to 7× pretraining speedup at fixed FLOPs ²⁸; and DeepSeek-V3 contributes the modern system stack — DualPipe for near-full computation-communication overlap across forward/backward chunks, auxiliary-loss-free load balancing, and fp8 at scale ³⁰. The open systems problem is expert communication: shared-memory fusion of MoE communication and computation is the current frontier, with the argument that coarse-grained overlap impairs compute efficiency ²⁹.

Communication: the scaling tax

Once the model fits and precision is low, wall-clock time is dominated by how much of the interconnect’s cost can be hidden. The end-to-end ceiling was set early: a communication backend combining ring allreduce, mixed precision, overlap, and lazy allreduce trained ImageNet/AlexNet 95 epochs in 1.5 minutes on 512 GPUs (410.2×) and ResNet-50 in 7.3 minutes (434.1×) ³¹ — on 56 Gbps-era networking, which is the point: communication software was the bottleneck, not the hardware.

The modern evidence refines this into three moves. Dedicated collective hardware: ACE, a collective-communication engine at the accelerator endpoint, cuts the memory bandwidth needed to drive the network by 3.5× and raises effective network bandwidth utilization by 1.44× (up to 2.67×) ³². Software overlap: CO2’s local-updating plus asynchronous communication achieves full overlap and a scalability ratio of 1.08 from 16 to 128 GPUs versus AdamW’s 0.93 ³⁴; FLUX fuses over-decomposed communication and compute kernels to overlap up to 96% of tensor-parallel communication, yielding up to 1.24× training speedup over Megatron-LM ³⁵; decomposition-based overlap for intra-layer model parallelism addresses the same problem at the schedule level ³³; and communication partitioning enables overlap in hybrid-parallel settings where fusion or op-scheduling alone fails ³⁷.

The honest counterweight is a 2025 characterization study: overlapping communication with compute causes an average 18.9% (up to 40%) compute slowdown versus non-interfered execution, and power capping severely exacerbates it — though overlap still nets an average 10.2% (max 26.6%) over sequential execution ³⁶. Overlap is not free; it is a trade whose value depends on hardware, precision, and power state. The record BERT case study shows the highest-value overlap trick inside the closed rules: gradient clipping before allreduce (bucket-wise) so synchronization overlaps gradient computation, rather than after ². One caution: compressed-communication systems like 1-bit LAMB ³⁸ are effective on low-bandwidth clusters but change the optimizer’s real-valued function via error feedback, and would need explicit working-group approval under the closed division’s numerical-equivalence rule — treat as non-compliant unless approved.

Data pipeline: the hidden third of wallclock time

The systems literature’s most repeated finding is that training is often stalled waiting for data. A Microsoft production study across nine models found training time frequently dominated by data-stall time — waiting for fetch and preprocessing ³⁹; locality-aware loading with software caches gives >30× data-loading speedup at 1,024 learners ⁴¹; and sharing one data loader across collocated processes raises throughput up to 100% and cuts cloud CPU cost by half ⁴⁰. The MLPerf community’s own storage benchmark was motivated by the same observation: datasets are outgrowing main memory, and I/O is the neglected axis ⁴².

The closed division makes this literature directly actionable because the rules draw a clean line: the dataset may be reformatted and preprocessed off-clock (only per-run random transforms are timed), but it must start on durable storage — explicitly not RAM — and caches must be flushed before a run. So the compliant data play is: aggressive offline preprocessing, sharded and packed formats, fast NVMe, and a pipeline deep enough that on-clock stalls vanish. The BERT case study quantifies the ceiling of one such trick: sequence-length packing (allowed as arbitrary batching, subject to RCP approval) has a theoretical 2× speedup available from load balancing alone, because unpadded BERT at length 512 wastes nearly half its compute on padding ².

Kernels and graphs: mathematically equivalent speed

The closed division explicitly permits graph transformations that are mathematically equivalent — dead-code elimination, common-subexpression elimination, recomputation — and the strongest example in the literature is attention. FlashAttention’s IO-aware tiling computes exact attention with linear memory and delivered a 15% end-to-end BERT-large speedup versus the MLPerf 1.1 training record, 3× on GPT-2, and up to 7.6× on the attention kernel itself ⁴³; FlashAttention-2’s parallelism rework reaches ~2× beyond that, up to 225 TFLOPS/s per A100 (50–73% of peak) ⁴⁴. Because the math is unchanged, these kernels are unambiguously inside the closed division — they are the template for what “same graph, faster execution” looks like, and the reference implementations of the v6.0 LLM benchmarks already consume them.

Benchmark-specific playbooks

BERT (v5.0, deprecated). The only fully documented closed-division optimization campaign: stratification-based presorting of the dataset plus bucket-wise clip-before-allreduce plus re-tuned Adam produced 25.1 s (22.3 s best run) on 1,024 A100s, 1.33×/1.57× faster than the top v1.1/v2.0 submissions ². Its transferable lessons: dataset-order engineering is legal (training data order must be “conceptually the same,” and stratification preserves the random-order contract), evaluation-adjacent scheduling matters, and network topology (200 vs 100 GB/s, SHARP presence) explained a 43.5 s vs 59.5 s gap on nominally identical GPUs.

DLRMv2 (v6.0). The model is embedding-dominated: tables of multiple GB each make data parallelism prohibitive, forcing model-parallel embedding sharding with personalized all-to-all after lookup ⁴⁵. The SC24 work adds the modern optimization set for the fixed-Adagrad closed benchmark: tensor-train decomposition of embeddings plus sorted-index micro-batching plus pipelined embedding-layer training to cut both compute and communication ⁴⁶. Since DLRMv2 trains at most one epoch and the LR schedule is fixed, this benchmark is the purest throughput race in the suite.

Llama2-70B-LoRA (v6.0). Fine-tuning with frozen weights and rank-decomposition adapters cuts trainable parameters 10,000×, GPU memory 3× (1.2 TB to 350 GB on GPT-3 175B), and gives a 25% training speedup because gradients for the vast majority of parameters are never computed ⁴⁷. For a benchmark whose reference already fixes LoRA alpha/rank/dropout, the remaining levers are the system ones: the model is small enough to data-parallel with aggressive overlap, and the evaluation schedule (every 384 sequences) rewards tight checkpointing.

MoE pretraining (GPT-OSS-20B, DeepSeek-V3-671B). Covered by the MoE and systems evidence above; the reference implementation (NeMo/Megatron-Bridge for DeepSeek) already embodies the DualPipe/fp8 stack ³⁰, so submitters compete on the margins: expert-parallel communication fusion ²⁹, load-balancing stability, and GBS choice within the fixed LR formula.

Where the evidence disagrees

Does large batch hurt generalization, and does it matter? Keskar et al. reported gaps up to 5% and the sharp-minima explanation ⁴; Goyal et al. showed no loss up to 8k with proper LR scaling and warmup ³; and the MLPerf-rules study found generic optimizers close the gap at even larger batches ⁶. The best resolution is chronological and methodological: the early result used fixed hyperparameters; once warmup and scaling were treated as first-class, the gap largely vanished — the sharp-minima story was an artifact of undertuned optimization, a position the later evidence supports.

Is LAMB special? You et al. reported Adam failing beyond batch 16,384 and LAMB fixing it ⁵; Dahl et al. did not reproduce the failure, finding tuned Adam beats LAMB at batch 65,536 ⁶. The disagreement is about tuning budget and metric (steps vs wall-clock): LAMB’s advantage was real on the TPU stack of its era with the tuning performed, and not reproducible as a law. For closed division the practical consequence is mild — the optimizer is fixed by the rules anyway — but it changes how much to trust reference hyperparameters.

Is communication overlap free? CO2 and FLUX present overlap as near-total recovery of the communication tax ^{34 35}; the ISPASS 2025 characterization measures a real compute cost (18.9% average, up to 40%) and finds the net gain shrinks to ~10% on average, with power capping turning it negative ³⁶. The disagreement dissolves once you separate measurement conditions: the negative side appears when compute and communication contend for the same execution resources (power-capped, fused-poorly), while the positive results come from systems engineered specifically to avoid contention (dedicated engines, kernel fusion, over-decomposition). The practical rule: overlap pays, but measure it on your exact hardware and power budget.

How much does batch size cost in epochs? The MLPerf paper’s ResNet-50 data shows epochs-to-target rising sharply with GBS ¹, while the critical-batch-size literature says the loss-per-token penalty only appears beyond the CBS plateau ⁷. These are consistent once you separate the regime: the MLPerf numbers are at 4K–16K where CNN training already exceeded the CBS; the CBS plateau is where a well-chosen GBS should sit. The tension is real only in that the closed division forces a single fixed GBS while the CBS picture is intrinsically dynamic.

Gaps and open questions

No published RCP playbook. The single most important strategic question — how to choose GBS and LR so that mean epochs-to-convergence lands just at the RCP boundary while throughput is maximized — appears in no retrieved paper. It is rules-level knowledge (the t-test normalization, interpolation between RCP points, score scaling) that the literature has not touched. What would settle it: an empirical study of epochs-to-target distributions across GBS for a v6.0 benchmark, which only the reference owners (or a well-funded submitter) can produce.

No v6.0-era submission analysis in the open literature. All published case studies are BERT-era (v1.1/v2.0) ². The Llama 3.1 405B, DeepSeek-V3, FLUX.1, and DLRMv2 rounds have no equivalent public dissection; the one recent systems account (AMD Instinct submissions) is paywalled and could not be retrieved. The field’s knowledge of what actually wins the current suite is therefore largely grey.

Compliance of emerging techniques is untested. Batch-size warmup ⁷ and fp4 training ¹⁴ are both promising and both sit in a gray zone: the former against RCP matching (fixed GBS), the latter against the “numerically safe” definition at scale. No working-group decision or public test exists for either in a closed-division setting. Similarly, compressed communication ³⁸ needs explicit approval that no submitter has documented.

Evaluation quantization is documented only in the rules. The v5.0 note that coarse evaluation spacing cost 12–16% when convergence landed just after an evaluation is a rulebook observation with no systems paper behind it. The interaction between evaluation frequency, GBS, and the timestamped-checkpoint stop rule is a measurable, publishable effect that would directly inform v6.0 scheduling.

Confidence and limitations

Confidence in the per-lever effects is moderate-to-high: the large-batch, precision, parallelism, and overlap results are replicated across independent groups and hardware generations, and the load-bearing numbers were verified against full texts in this review. Confidence in the overall “optimal closed-division strategy” is lower, because no source directly tests it — the synthesis is an inference from the rules (retrieved and read in full) plus adjacent systems evidence. Limitations of this review: Semantic Scholar contributed nothing (rate-limited), so coverage leans on OpenAlex/Crossref/arXiv; five sources are abstract-only because of paywalls or bot-blocking (most load-bearing claims still rest on full texts); the v6.0-specific literature is thin by the field’s own youth; and the review excludes non-English work and inference-benchmark transfer (which is a real but separate literature). One source with no retrievable abstract was dropped rather than cited on title alone.