Lewis Won. Literature Reviews
On this page

Global batch size and convergence in large language model training

How does global batch size affect the convergence and compute efficiency of LLM pretraining?

Updated
10 Aug 2026
Sources
33
Years
2017–2026
Confidence

https://reviews.lewiswon.me/reviews/batch-size-convergence-llm/ · Updated 10 Aug 2026

Download Markdown

batch sizeLLM pretraininggradient noise scalecritical batch sizeconvergence

How this review was made
Databases
OpenAlex, arXiv, Crossref, Semantic Scholar (rate-limited, did not answer)
Queries (literal)
batch size convergence large language model training
large batch training generalization gap deep learning
gradient noise scale critical batch size
linear scaling rule learning rate batch size
optimal batch size neural network training
batch size language model pretraining compute efficiency
large batch stochastic gradient descent sharp minima
batch size learning rate scaling deep learning convergence
global batch size distributed training convergence
minibatch size generalization convergence
optimal batch size large language model pretraining
batch size loss spikes language model training
effective batch size gradient accumulation
batch size Adam optimizer convergence
critical batch size scaling law
batch size schedule learning rate schedule
large batch training language models convergence
batch size compute optimal training tokens per step
Search last run
2026-08-10
Screening
33 sources used · 2017–2026 · standard review

Summary

The short version

The global batch size — the number of tokens per optimizer step — is not a dial that changes where an LLM converges, but a dial that changes how it gets there: it trades three currencies against each other (number of steps, number of tokens seen, wall-clock time). A measurable statistic called the gradient noise scale marks a critical batch size (CBS) beyond which larger batches buy almost no additional convergence speed, and the efficiency curve is smooth and well understood. The evidence, from the 2018 noise-scale framework through to 2024–2026 scaling-law studies, says the CBS grows over the course of training and with dataset size — not, contrary to earlier folklore, with model size — and that learning-rate scaling rules are what make large batches safe. Every major LLM pretraining run since GPT-3 has therefore ramped its batch size upward during training. Confidence is high for the qualitative framework and production practice; the exact scaling exponents are newer and come from single groups, so those carry less weight.

Why this question

For anyone provisioning or scheduling LLM training, batch size is the single most consequential knob that is not a model hyperparameter. On a fixed GPU fleet, global batch size is the product of data-parallel degree, micro-batch size, gradient-accumulation steps, and sequence length — so choosing it is simultaneously a parallelism decision (how many GPUs can work on one step), a throughput decision (tokens per second), and an optimization decision (loss per token). Infra teams routinely see “scale out data parallelism” as free speed, but if the batch is already past the critical size, doubling it adds GPUs that buy nothing but a shorter step interval.

The question also carries a persistent folk belief: that batch size should grow with model size (a reading of early scaling-law work), and that large batches hurt final quality (“the generalization gap”). The recent literature has substantially revised both. Getting the relationship right — and knowing where it is still contested — is exactly what separates a training plan that reaches a target loss at minimum cost from one that burns GPU-hours on a batch that is too large to help or too small to saturate the hardware.

Scope and methods

Question. How does the global batch size affect convergence — steps-to-loss, tokens-to-loss, and final loss — for autoregressive language model pretraining, and how should it be chosen for compute-efficient training?

Inclusion. (a) Foundational theory and empirical studies of batch size in deep learning (gradient noise scale, learning-rate scaling rules, generalization) from 2016 onward; (b) pretraining reports from major LLM efforts that state their batch configuration; (c) 2023–2026 studies specifically about batch size in LLM pretraining. Exclusion. Batch size in inference or serving, RL training batches, federated learning, batch scheduling in operations research, and CNNs-only studies without a generalizable claim (except the foundational ones above).

Searches. 18 concept queries were run on 2026-08-10 across OpenAlex, arXiv, Crossref, and Semantic Scholar (which never answered — rate-limited from this IP), producing 1,504 pooled records, deduplicated to ~1,100, topic-screened to ~60 candidates, of which 33 were included. Every included source was retrieved in-session (31 full texts, 2 abstracts), every DOI was verified (Crossref for publisher DOIs, abs-page checks for arXiv DOIs), and load-bearing numbers were re-verified against full texts by grepping the source documents. One remembered landmark (“Large Batch Optimization for Deep Learning: New Bounds and Complexity”) could not be retrieved and was dropped rather than cited from memory. Exact queries are recorded in the frontmatter.

The landscape

The literature splits into four eras that still coexist. (1) CV-era foundations (2016–2018): the generalization-gap debate and the scaling rules — Keskar, Goyal, Smith & Le, Jastrzębski — mostly on ImageNet/CIFAR. (2) The noise-scale framework (2018): McCandlish et al. gave the field a quantity to measure, and OpenAI’s scaling-law papers embedded it in a power-law world. (3) LLM production practice (2020–2024): GPT-3, PaLM, Chinchilla, Llama 2/3, DeepSeek-V3 report batch sizes in tokens per step and — increasingly — batch ramps, but rarely ablate the choice. (4) Batch-specific LLM studies (2024–2026): at least five independent groups (Harvard/Amazon, Cerebras, Huawei, Ai2, and systems papers on adaptive batching) have now measured CBS scaling or built batch schedulers directly for pretraining-scale models. The shape of the field: the theory is mature and remarkably stable; the quantitative scaling laws are young and still being reconciled with each other.

Theme 1 — Convergence has three currencies, and batch size exchanges them

The central result, from McCandlish et al., is that training has a minimum possible number of steps S_min and a minimum possible number of tokens E_min, and the batch size trades one against the other along a smooth hyperbolic curve 5. Their empirically motivated model gives the loss reduction per step as ΔL = ΔL_max / (1 + B_noise/B), where B_noise is the gradient noise scale — the trace of the gradient covariance (roughly, tr(Σ)/|G|² in the well-conditioned case) divided by the squared gradient norm 5. Below the noise scale, each step is dominated by noise and steps-to-target falls linearly as batch grows; above it, the gradient is already well averaged and extra data adds almost nothing. The critical batch size B_crit — the batch at which time and compute efficiency are balanced — is empirically ≈ B_noise 5.

Three practical consequences follow. First, a batch below B_crit wastes parallelism: the same loss could be reached in fewer steps with a bigger batch, at equal token cost. Second, a batch far above B_crit wastes data: tokens are consumed with no convergence benefit. Third, and most important for schedulers: the noise scale grows as the loss falls 5 — the same batch that is compute-efficient early becomes data-wasteful later, which is the theoretical justification for batch ramping. Kaplan et al. confirmed the same structure for language models, fitting the critical batch size as a pure power law in the loss, B_crit(L) = B*/L^(1/α_B) with B* ≈ 2×10⁸ tokens and α_B ≈ 0.21, and predicting roughly 1–2 million tokens per step at convergence for the largest models they could train 9. Henighan et al. added the caveat that a batch too large early in training wastes compute, and used the critical-batch concept to correct their cross-domain scaling estimates 10.

The systems implication is direct: Megatron-LM’s scaling study — 8.3B-parameter GPT-2 at 512×1024-token batches on 512 GPUs, 15.1 PFLOP/s at 76% efficiency — is precisely the regime where the batch is the thing being scaled with data parallelism 18, and the paper explicitly frames large-batch optimization difficulty as the obstacle to weak scaling 18.

Theme 2 — The learning-rate scaling rules that make large batches safe

The CV era established that large batches fail for a recoverable reason: the learning rate must move with the batch. Goyal et al. showed on ImageNet that multiplying the minibatch by k and the learning rate by k (the linear scaling rule), plus a warmup phase, trains ResNet-50 at batch 8192 on 256 GPUs in one hour at the same top-1 accuracy as the batch-256 baseline — and argued the failure mode is optimization difficulty, not generalization 2. Smith & Le gave the mechanism: SGD behaves like a stochastic differential equation whose “noise scale” is g ≈ εN/B, so with the learning rate fixed there is an optimal batch B_opt ∝ εN 3. Their follow-up is the cleanest statement of the trade: don’t decay the learning rate, increase the batch size — whenever the LR schedule would drop by a factor α, multiplying the batch by α reproduces the same learning curve with fewer updates, and the rules B ∝ ε and B ∝ 1/(1−m) (momentum) hold across SGD, momentum, Nesterov, and Adam 4. Jastrzębski et al. independently showed the learning-rate-to-batch-size ratio is the control parameter of SGD dynamics, with the width of the reached minimum — and hence generalization — set by ε/B 8.

The generalization gap itself is real but contingent. Keskar et al. documented up to a 5% test-accuracy drop for large batches, attributing it to convergence to sharp minima 1. The subsequent literature reads this as a failure to rescale hyperparameters: when LR scaling is applied, the gap largely vanishes (Goyal on ImageNet) 2, and in the LLM era, small-batch and large-batch training reach equal loss at equal tokens when each is tuned 2822. Leslie Smith’s practice-oriented work — the 1cycle schedule and LR range test — encodes the same insight operationally: larger batches help, with tapering returns, provided the LR is raised proportionally 7.

For Adam-family optimizers the rules are more delicate. Li et al. prove and demonstrate a “surge” phenomenon: for Adam, the optimal learning rate first rises then falls as the batch grows, peaking near the noise scale, with the peak shifting toward larger batches as training progresses 19. This non-monotonicity is why “linear scaling” is a poor rule of thumb for AdamW and why labs retune LR per batch. Optimizer choice also sets the ceiling: AdamW stops scaling past batch 16K on BERT-Large, while LAMB’s layer-wise adaptive rates reach 65K with no accuracy loss (76-minute BERT-Large) — and going from 65K to 131K buys nothing 32. Theory on the Adam side is catching up: mini-batch and distributed Adam provably achieve linear speedup in batch size 12, and for SGD with momentum the batch saturates at the “implicit conditioning ratio,” beyond which convergence matches full-batch momentum at O(1/√κ) 13. At the floor, the gradient-estimator variance provably decreases with batch size 30, and empirically batch-1 training is stable when every other hyperparameter is retuned 28.

Theme 3 — Production practice: tokens per step across the LLM era

Every major pretraining run now publishes its batch in tokens per step, and the trajectory is a monotone climb with compute — with ramping as the norm. GPT-3 ramped the batch linearly from 32K tokens to its full value (0.5M for the small models up to 3.2M for 175B) over the first 4–12B tokens of training 11. Megatron-LM trained its GPT-2 family at 512×1024 tokens per step for 300K iterations 18. Chinchilla (70B) ran 1.5M tokens per step, doubling to 3M midway — with batch choice attributed to McCandlish-style heuristics rather than ablation 14. PaLM 540B staged the batch at 1M → 2M → 4M tokens (at steps 50K and 115K), explicitly reasoning that smaller batches are more sample-efficient early and larger batches are needed later to keep hardware fed 15. Llama 2 held a fixed 4M-token global batch for all sizes across 2T tokens 16. Llama 3 405B doubled 4M → 8M → 16M tokens (with sequence length 4096 → 8192), stating the motivation baldly: “a lower batch size early in training to improve training stability, and increase it subsequently to improve efficiency” 25. DeepSeek-V3 ramped from 3,072 to 15,360 sequences — roughly 13M to 63M tokens per step at its 4K sequence length — over the first 469B tokens, then held 24.

The small-scale end tells the same story from the other side: when compute is scarce, batch size is still a first-order knob. Cramming, training BERT in one day on a single GPU, found the 96-token micro-batch was far below optimal — 1,536 tokens minimized pretraining loss and 4,032 maximized downstream accuracy, recovered via gradient accumulation 33. The takeaway across scales: the “right” batch is not a model property; it is the largest batch that does not yet waste tokens, which is why production runs ramp into it.

Dynamic batching is the automation of this ramp. Monitoring the gradient direction and accumulating until it starts to fluctuate beat a fixed 25K-token batch by +0.73/+0.82 BLEU on WMT14 En-De/En-Fr 17. Norm-test adaptive schedules (choose the next batch from the gradient norm ratio) beat constant and heuristic-warmup baselines on MicroLlama, TinyLlama, and OpenLlama with convergence guarantees for Adam 23. Cerebras’ noise-scale tracking — per-example gradients from the normalization layers predict the full gradient noise scale at zero throughput overhead — drove a batch schedule that cut wall-time 18% on a Chinchilla-optimal model 20.

Theme 4 — Scaling laws for the batch size itself

The 2024–2026 studies converge on a claim that revises the folklore: the critical batch size scales with the dataset, not the model. The observation is older than the laws: Hestness et al. noted as early as 2017 that larger models and larger training sets get harder to optimize, and that raising batch size and learning rate closed a significant portion of the gap to the power-law trend 6. Zhang et al., sweeping models from 85M to 1.2B parameters on C4 with careful control of LR and momentum, found CBS ≈ 93.20·N^0.47 under Chinchilla-optimal token counts, but only N^0.087 with data size fixed — while scaling data at fixed model size gives B* ≈ 22.91·D^0.47 21. Their headline: CBS scales primarily with data size, a finding they justify theoretically through infinite-width limits 21. Cerebras’ Power Lines study, fitting over many (N, D) pairs, agrees: both B_opt and B_crit follow power laws in total training tokens D independent of N, with B_crit ≈ 0.0471·D^0.47 under their definition (the coefficient differs from Zhang’s partly because the papers define “critical” at different points on the efficiency curve — a factor of 5 by their own conversion — though the exponents agree, 0.47 vs 0.462) 27. Power Lines additionally derives a scaling law for the AdamW timescale τ = B/(ηλD), whose optimal value is a power law in the tokens-per-parameter ratio D/N — a practical recipe for picking LR, weight decay, and batch jointly 27. Shuai et al. fit batch size into the loss function itself: on the compute-efficient frontier B_opt ≈ 6.42×10³·C^0.102, and with N and D fixed B_opt ≈ 3.24×10³·D^0.264 — about 4.7M tokens per step for a 1T-token run 22.

These laws reconcile with the older ones if the loss is the mediating variable: McCandlish and Kaplan said B_crit is a power law in the loss 59, and the loss at convergence is set by the data budget D — so “batch scales with data” and “batch scales with loss” are the same statement at fixed D/N. The model-size folklore came from conflating D/N with N.

Theme 5 — Batch size as a systems knob, and where the loop closes

For the infra view, the global batch is not chosen in isolation: it is the product of data-parallel degree, micro-batch, gradient accumulation, and sequence length, and it determines both MFU and convergence. Llama 3’s own scaling table shows the coupling — holding tokens per batch constant at 16M while moving from 8K to 16K GPUs drops the per-DP batch and costs MFU (43% → 41%) 25. PaLM makes the same point in reverse: 4M tokens per step was already large enough that the authors doubted larger batches would stay sample-efficient 15. The recent systems work attacks the loop directly: COPUS co-adapts batch size, micro-batch, and 3D parallelism online using gradient-noise estimates, saving 3.9–8.0% time-to-convergence (peak 11.1%) on 3B–32B models 31; AdLoCo adapts local batch sizes in low-communication (DiLoCo-style) training, reaching lower perplexity in fewer steps than the fixed-batch baseline 29. These are small numbers next to the 2–3× wall-time swings that a mis-chosen batch can cause at either extreme, but they are the first systems results that treat the batch as an online control variable rather than a fixed hyperparameter.

Where the evidence disagrees

Does the generalization gap exist? Keskar’s sharp-minima gap (up to 5%) 1 and Goyal’s “it’s just optimization” rebuttal 2 are both from 2017–18 and both correct in their own regime: the gap shows up when the LR is not rescaled, and disappears when it is. The LLM-era studies side with Goyal — equal loss at equal tokens with tuned hyperparameters 2822 — but with an important residue: small batches are more robust to optimizer and hyperparameter choices 28, which is why they survive in research settings and why the “gap” keeps being rediscovered in under-tuned runs.

What drives the critical batch size: loss, data, or model? McCandlish/Kaplan frame it as a function of the loss 59; Zhang and Cerebras frame it as a power law in D, essentially independent of N 2127. These are compatible (loss at convergence is a function of D), but the practical prescriptions differ: the first says “measure the noise scale of your run,” the second says “read it off your data budget.” The quantitative disagreement between Zhang and Cerebras — coefficients differing by ~19% after conversion, definitions of “critical” differing by 5× — is a measurement-consensus gap, not a contradiction, and both agree the exponent in D is ≈ 0.47.

Is the gradient noise scale a trustworthy proxy? McCandlish’s framework is the theoretical backbone and was used explicitly by GPT-3-era practitioners 511. But Ai2’s direct measurements found the noise scale underestimates the CBS, calling it unreliable as a proxy 26, while Cerebras found it practical enough to schedule a whole run on, at zero overhead 20. The disagreement is partly about what “critical” means and partly about estimation error at the point where the curve flattens — the region where any estimator is least certain.

Ramp, hold, or adapt? GPT-3, PaLM, Llama 3, and DeepSeek-V3 ramp 11152524; Chinchilla doubles once 14; Llama 2 holds fixed 16; Smith & Le say a ramp is mathematically equivalent to an LR decay schedule 4; Merrill says batch warmup is the natural use of CBS measurements 26; Lau says adaptive beats heuristic warmup 23. There is no dispute that batch should grow over training — only about whether the growth is a stability measure, a data-efficiency measure, or an LR-schedule artifact, and whether it should be scripted or measured.

Gaps and open questions

The most important gap is scale: the CBS scaling exponents come from models ≤ ~1.2B parameters (Zhang) or Cerebras’ proxy-scale sweeps, and the production datapoints (Llama 3’s 16M, DeepSeek’s 63M) are single runs, not ablations. Whether B_crit ∝ D^0.47 holds at 100B+ parameters and 15T tokens is an extrapolation. Second, MoE changes the unit of analysis: DeepSeek-V3’s 63M-token batch is a per-expert batch of only ~1-2M tokens after routing, so “batch size” for convergence purposes is not the global token count 24 — nobody has published an expert-level CBS. Third, the joint law linking batch, LR schedule, and weight decay is only beginning to be formalized (Power Lines’ τ is the first serious attempt) 27. Fourth, sequence-length changes during training (Llama 3 doubled seqlen with its batch 25) shift the token-per-sequence ratio in ways the D-based laws do not model. Fifth, the interaction between batch size and data repetition is unexplored in the batch-specific literature. Finally, all of the “optimal batch” laws optimize for training FLOPs; with inference dominating total cost, the batch that minimizes lifetime cost may be smaller than the training-compute-optimal one 22.

Confidence and limitations

High confidence attaches to: the noise-scale framework (replicated across domains and still the working theory in 2026) 59; the LR-scaling rules 24; the production batch numbers (verified directly in the primary sources) 1115162524; and the claim that CBS scales with data, not model size (two independent groups, consistent exponents) 2127. Moderate confidence attaches to the specific coefficients (single papers, definitions not yet standardized) and to the adaptive-batching systems results (preprints, small speedups) 312923.

Limitations of this review: it is English-language and preprint-heavy for the recent work; all 33 sources were read in full text or verified section-by-section, but two contribute only contextual batch content (Hestness’s and Henighan’s scaling studies mention batch size in passing rather than as a controlled variable); the production reports are self-reports with no counterfactual ablations; and the review deliberately excludes the RL and fine-tuning batch-size literatures, which have their own dynamics. One remembered landmark could not be retrieved and was dropped rather than cited. Search was run on 2026-08-10; the field is moving quickly enough that the CBS-exponent consensus could shift.

Jump to references ↓

Evidence table

keydesignsamplemeasurefindinglimitationsconfidenceaccessnote
bergsma2025powerlinesbenchmark~400 LLMs from 111M to 3.3B parameters trained at 20-1280 tokens-per-parameter (datasets 2.19B-548.6B tokens) on Cerebras CS-3 systems; batch sizes swept with weight decay tunedValidation loss vs batch size; optimal (Bopt) and critical (Bcrit) batch size scaling exponents; AdamW timescale tau=B/(eta*lambda*D)Both Bopt and Bcrit scale as power laws in total training tokens D independent of model size N (R^2=0.984 and 0.940; Bcrit ~ 0.0471*D^0.47, exponent 0.47 vs 0.462 in zhang2024how; Bopt exponent 0.37-0.39, Bcrit 0.49-0.53 10th-90th percentiles), while optimal tau obeys tau_opt proportional to (D/N)^-0.5 and optimal lambda scales linearly with B up to Bcrit.Models only up to 3.3B parameters; exponents carry uncertainty (percentile ranges); Bcrit estimation depends on proper weight decay tuning; hardware-specific (Cerebras).highfull-textReframes Bopt/Bcrit scaling from compute/loss (Kaplan) to dataset size D — central modern evidence for how global batch size should grow with data.
brown2020languagecase-studyGPT-3 family: 8 autoregressive Transformers from 125M to 175B parameters, all trained on 300B tokens, context 2048, batch sizes 0.5M-3.2M tokensFew-shot task performance; batch size and learning rate choices per model sizeBatch size was ramped linearly from 32k tokens to the full value (0.5M for small models up to 3.2M tokens for 175B) over the first 4-12B tokens of training, guided by measured gradient noise scale, with larger models using larger batches and smaller learning rates (6.0e-4 down to 0.6e-4) plus 375M-token linear LR warmup and cosine decay to 10% over 260B tokens.Single model family; batch schedule is a heuristic chosen via GNS rather than controlled ablation; authors acknowledge data contamination issues from web corpora.highfull-textCanonical LLM case of batch-size warmup (32k to 3.2M tokens) and GNS-guided batch sizing that later LLMs emulated.
chen2021towardstheoreticalTheory for smooth non-convex stochastic optimization; experiments: LeNet on MNIST and ResNet on CIFAR-100 with different batch sizesIteration complexity to reach epsilon-stationary point as a function of mini-batch size sMini-batch Adam reaches an epsilon-stationary point in Omega(epsilon^-4 * s^-1) iterations with batch size s, i.e., linear speedup in batch size (same order as mini-batch SGD), and distributed Adam achieves linear speedup in number of workers under the parameter-server model.Assumes smoothness and bounded gradient variance; experiments are small-scale (MNIST/CIFAR-100) and confirm theory qualitatively rather than at LLM scale.highfull-textProvides the theoretical linear-speedup basis for why increasing global batch size reduces steps-to-convergence for Adam.
chowdhery2022palmcase-studyPaLM: 8B, 62B, and 540B dense Transformers trained identically except batch size on 780B tokens, 6144 TPU v4 chipsFew-shot benchmark performance; training throughput and MFU; batch size schedulePaLM 540B used a staged batch schedule of 512 sequences (1M tokens) until step 50k, 1024 (2M tokens) until step 115k, then 2048 (4M tokens) until step 255k, explicitly because smaller batches are more sample-efficient early while larger batches give better gradient estimates later, reaching 238.3K tokens/sec (46.2% MFU) at batch 2048.Single training run without controlled batch-schedule ablations; loss spikes occurred and were mitigated by restarting from earlier checkpoints.highfull-textProduction-scale demonstration of a 1M-4M token batch ramp with stated sample-efficiency rationale (citing Smith 2018, McCandlish 2018).
deepseek2024v3case-studyDeepSeek-V3: 671B-parameter MoE (37B active) pretrained on 14.8T tokens, 2048 H800 GPUs, sequence length 4KTraining stability (loss spikes, rollbacks), cost in GPU hours, downstream benchmark performanceBatch size was gradually increased from 3072 to 15360 sequences over the first 469B tokens then held at 15360, with LR linearly warmed up to 2.2e-4 over 2K steps, held constant to 10T tokens, then cosine-decayed to 2.2e-5 over 4.3T tokens — the run reported no irrecoverable loss spikes and cost 2.788M H800 GPU hours total.Report-style paper with no controlled comparison of batch schedules; batch size reported in sequences not tokens; stability also attributed to FP8/auxiliary-loss-free design choices.highfull-textFrontier-scale batch ramp (3072 to 15360) paired with aggressive LR schedule as evidence that large global batches are compatible with stable convergence.
dubey2024llama3case-studyLlama 3: 8B, 70B, and 405B dense Transformers; 405B pretrained on 15.6T tokens on up to 16K H100 GPUsTraining stability and loss; downstream benchmark quality; MFULlama 3 405B used a batch schedule of 4M tokens (seq len 4096) doubled to 8M tokens (seq 8192) after 252M tokens and doubled again to 16M tokens after 2.87T tokens, with 8,000-step linear warmup and cosine LR decay from 8e-5 to 8e-7 over 1.2M steps — reported as very stable with few loss spikes and no divergence interventions (smaller scaling-law runs used fixed batches of 250K-4M).Descriptive recipe rather than ablation; batch schedule motivated by stability/efficiency, not shown optimal; 8B/70B use fixed batches per compute scale.highfull-textState-of-the-art example of doubling batch size (4M-8M-16M tokens) in tandem with sequence-length warmup at 405B scale.
geiping2023crammingbenchmarkBERT-base crammed on a single GPU (GTX 2080 Ti / A4000) for one day; micro-batch 96 on most configspretraining MLM loss and downstream MNLI/SST-2 accuracy within a fixed compute budgetThe optimal batch size is several times the 96-token micro-batch: 1,536 for minimal pretraining loss and 4,032 for maximal downstream performance (gradient accumulation every 16 and 42 steps respectively).BERT-scale models and one-day budgets; conclusions may not transfer to large-scale pretraining.highfull-textEvidence that batch size is a first-order hyperparameter even in tiny-compute settings, and that micro-batch-limited hardware can compensate via gradient accumulation.
goyal2017accuratebenchmarkResNet-50 on ImageNet (256 to 8192 minibatch, 8 to 256 GPUs) plus Mask R-CNN detection/segmentation tasksTop-1 validation error vs minibatch size; training time; scaling efficiencyWith the hyperparameter-free linear scaling rule (multiply LR by k when minibatch is multiplied by k) plus a 5-epoch warmup, ResNet-50 trained with minibatch 8192 on 256 GPUs in 1 hour matches small-batch (256) accuracy with ~90% scaling efficiency from 8 to 256 GPUs — no accuracy loss up to ~8k minibatch, beyond which accuracy degrades.ImageNet/ResNet-50 focus; linear scaling breaks down in early training without warmup and beyond ~8k minibatch; Batch Normalization semantics change with batch size.highfull-textFoundational empirical result: large global batches preserve convergence/generalization if LR scales linearly and warmup is used.
gray2024normalizationbenchmarkTransformer LMs including 111M-parameter Chinchilla-optimal (Cerebras-GPT recipe) models on OpenWebText, replicated on NVIDIA A10 GPUs; GNS measured per layerGradient noise scale (GNS) estimation accuracy and cost; wall-clock training time under a GNS-guided batch size scheduleTotal GNS of contemporary transformers is predicted well by normalization-layer GNS alone, enabling zero-overhead per-example gradient norm computation (custom LayerNorm kernel), and a GNS-guided batch size schedule reduced wall-clock training time by 18% on a Chinchilla-optimal 111M LM.Case study at 111M scale only; authors report bf16/flash-attention numerical instability in replications; batch size changes did not affect measured GNS as the McCandlish model predicts in one replication.highfull-textMakes McCandlish-style GNS-based batch scheduling practical at zero throughput overhead — direct operationalization of critical-batch-size theory.
henighan2020scalingbenchmarkAutoregressive Transformers across four domains: image (YFCC100M ~1e8 images), video (~7e5 videos), multimodal image-text, and mathematical problem solvingCross-entropy loss L(x) = L_inf + (x0/x)^alpha_x vs model size, compute, dataset; optimal model size Nopt(C)Loss follows power-law-plus-constant scaling in compute, model size, and data across all four domains with optimal model size Nopt(C) proportional to C^0.7 nearly universal across modalities, implying dataset size D proportional to N^0.4 for compute-optimal single-epoch training (batch-size-specific findings are not central to this paper).Batch size is not a studied variable (fixed near-critical batches used); exponents are modality-dependent; fitted constants domain-specific.highfull-textEstablishes universal compute-scaling exponents that batch-size scaling laws (Kaplan's Bcrit, tokens-per-step) are embedded in.
hestness2017deepbenchmark6 deep neural network model families across 4 domains: machine translation, language modeling (LSTM/RHN word and char LMs), image classification, and speech recognitionGeneralization error vs training set size (learning-curve power-law exponents beta_g); model size vs data sizeGeneralization error follows power laws epsilon(m) ~ alpha*m^beta_g with empirical exponents settling between -0.07 and -0.35 (e.g., -0.0656 +/- 1% for word LMs), with model size scaling sublinearly in data; larger training sets/models become harder to optimize, and increasing batch sizes and learning rates typically closed a significant portion of the gap to the power-law trend.Exponents unexplained by theory; batch size treated only as an optimization knob in methodology, not a controlled variable; domain-specific constants.highfull-textData-scaling laws showing batch size/LR must grow with dataset scale to stay on power-law generalization curves.
hoffmann2022trainingbenchmarkOver 400 transformer LMs from 70M to over 16B parameters trained on 5B to 500B+ tokens; validation on Chinchilla 70B vs Gopher 280B, GPT-3 175B, Jurassic-1 178B, MT-NLG 530BFinal pretraining loss L(N,D) as a function of parameters and tokens; compute-optimal N and D allocationFor compute-optimal training, model size and training tokens should scale equally (doubling N requires doubling D, i.e., ~20 tokens per parameter), and the predicted compute-optimal Chinchilla (70B, 1.4T tokens) trained with batch size doubled from 1.5M to 3M tokens midway (Gopher used 3M to 6M) uniformly outperformed Gopher and other larger models (67.5% MMLU, >7% over Gopher).Batch sizes fixed by heuristic (doubling midway) rather than optimized; fits from models up to 16B; authors note McCandlish found only weak dependence of optimal batch size on model size.highfull-textChinchilla tokens-per-parameter law defines the data axis against which batch size (tokens per step) schedules are set in modern LLM training.
jastrzebski2017threetheoreticalSDE approximation of SGD; experiments with VGG11 and ResNet-56 (with BatchNorm) on CIFAR-10, batch sizes 50-640, learning rates 0.001-1.2Width of final minima (trace of Hessian), generalization, and training dynamics vs learning-rate/batch-size ratio (eta/S)The learning-rate-to-batch-size ratio eta/S is the key control parameter of SGD dynamics (it is the noise coefficient in the underlying SDE; expected loss at the minimum scales as (eta/S)*Tr(H)), so higher eta/S converges to wider minima with often better generalization, and cyclic batch-size schedules can replace cyclic learning-rate schedules with matching train/test curves.SDE derivation relies on CLT, quadratic-bowl and C=H assumptions valid mainly near convergence; experiments on small vision models only.highfull-textTheoretical foundation for treating batch size as a scheduleable analogue of learning rate (basis for batch ramps).
kaplan2020scalingbenchmarkTransformer LMs from 768 to 1.5B non-embedding parameters on WebText2 (22M to 23B tokens), trained 2.5e5 steps at batch 512 sequences of 1024 tokens with AdamCross-entropy loss power laws in N, D, C; critical batch size Bcrit(L); optimal batch sizeThe critical batch size obeys Bcrit(L) = B*/L^(1/alpha_B) with B* ~ 2e8 tokens and alpha_B ~ 0.21 — a power law in the loss alone — and the ideal batch size is roughly 1-2 million tokens at convergence for the largest models, while loss scales as power laws (alpha_N ~ 0.076, alpha_D ~ 0.095) over more than six orders of magnitude.Bcrit inferred from gradient noise scale rather than direct batch sweeps; WebText2-specific constants; the N/D allocation was later revised by Hoffmann et al. (Chinchilla).highfull-textOrigin of the critical-batch-size power law (Bcrit ~ L^-1/alpha_B) and the ~1-2M token optimal batch estimate used by GPT-3.
keskar2017onlargebatchbenchmarkMLPs (F2) and convnets (C1 etc.) on MNIST, CIFAR-10, CIFAR-100; large batch = 10% of training data vs small batch = 256, ADAM optimizer, 5 random restartsGeneralization gap (train-test accuracy difference) and sharpness of converged minima (Hessian eigenvalue spectrum)Large-batch methods show a generalization gap of up to 5% even for small networks because they converge to sharp minimizers, while small-batch methods converge to flat minimizers (with degradation emerging beyond a batch-size threshold near ~15,000 in their experiments), and warm-starting large-batch training from a small-batch solution can close the gap.Sharp-vs-flat distinction is contested (Dinh et al. show sharp minima can generalize); threshold batch size is dataset-specific; LB=10% of data conflates batch size with data fraction.highfull-textCanonical evidence for the large-batch generalization gap and the sharp-minima explanation motivating adaptive batch schedules.
kutuzov2025adlocoframeworkMicroLlama 300M pretrained on English C4 subset, 2400 steps, per-device batch 100 with 5-step gradient accumulation, AdamW lr 4e-4, 4 simulated GPUs on one A100Perplexity vs steps, communication count, wall-clock convergence; theoretical communication complexity boundAdLoCo (DiLoCo + norm-test adaptive local batch sizes + trainer merging + gradient-accumulation switch when requested batch exceeds 2x hardware max) reaches lower perplexity in fewer training steps than DiLoCo with communication complexity E[C(N)] = O((bmax*eta^2*L*(1+eta^2)*(F(x0)-F(x*))/sigma^2) * ln N), though exact speedup magnitudes appear only in figures.Single small model; multi-GPU cluster simulated on one GPU; quantitative speedups reported in figures rather than text; short 2400-step training.moderatefull-textApplies adaptive batch sizing to distributed low-communication LLM training, showing batch growth improves both convergence speed and communication efficiency.
lau2024adaptiveframeworkMicroLlama 300M (DDP-Norm), TinyLlama 1.1B and OpenLlama 3B (FSDP-Norm) pretrained on C4 (Llama 2 tokenizer); baselines: constant batches 2048/4096/8192 and stagewise 2048-4096-8192 (2.5-2.5-95%) warmupTraining/validation loss vs samples, gradient steps, wall-clock time, averaged batch size under norm-test adaptive schedules with AdamNorm-test adaptive batch schedules beat constant batch sizes and heuristic warmup schedules on MicroLlama 300M, TinyLlama 1.1B and OpenLlama 3B, with theoretical convergence guarantees for Adam.Proofs-of-concept with limited training (batch sizes hit the cap early); OpenLlama 3B run at reduced sequence length 512; norm-test tolerance eta is an extra hyperparameter.highfull-textPrincipled, theory-backed adaptive batch schedules (with Adam convergence guarantees) for LLM pretraining — key modern evidence on dynamic global batch sizing.
lee2022trajectorytheoreticaln/a (theory): mini-batch SGD with momentum (SGD+M) on least squares with large sample count n and dimension d, Gaussian random data, constant learning rateconvergence rate of SGD+M as a function of batch fraction (loss trajectory characterized by a discrete Volterra equation)When the batch fraction zeta >= ICR (implicit conditioning ratio = kappa_bar/kappa), SGD+M converges linearly at rate O(1/sqrt(kappa)) matching optimal full-batch momentum, and there is a saturating batch fraction (zeta = ICR, before full batch) beyond which increasing batch size does not improve the rate; for zeta < ICR the rate is zeta/((1-zeta)*kappa_bar), i.e. no acceleration over single-batch SGD.Linear least-squares setting only; constant learning rate; results are in the large-n, large-d limit; no generalization analysis.highfull-textGives a rigorous definition of large- vs small-batch regimes for momentum SGD and a provable batch-size saturation point (ICR) tied to Hessian spectra.
li2024surgetheoreticalTheory (quadratic loss approximation) + experiments: 5-layer CNN on Fashion-MNIST, ResNet-18 on Tiny-ImageNet, DistilGPT2 on ELI5-Category, MoE (Mistral-MoE-like) on RedPajama-v2; Adam with beta1/beta2 in {0, 0.9/0.999}; batch-size grid ranges ~1-1164 (CNN), 4-114 (DistilGPT2), 16-376 (ResNet18), 192k-12M tokens (MoE)Optimal learning rate as a function of batch size for Adam-style (sign-of-gradient) optimizersFor Adam-style optimizers the optimal learning rate first rises then falls as batch size increases ('surge'), peaking near the gradient noise scale B_noise (with the peak moving to larger batches as training progresses), in contrast to SGD where epsilon_opt(B) ~ (epsilon_max/B_noise)*B scales linearly for B << B_noise.Theory based on quadratic approximation of the loss; authors note higher-order terms may be needed at large learning rates; validated on 4 workloads only.moderatefull-textShows the LR-vs-batch-size scaling rule differs fundamentally between Adam-style and SGD-style optimizers, relevant for large-batch LLM training with Adam.
marek2025smallbatchbenchmarkTransformer decoder-only 30M-param model on 600M tokens of FineWeb-Edu; GPT-2 (124M) and GPT-3 (1.3B) on FineWeb; Gemma 3 (4B) fine-tuning; batch sizes spanning {1, 4, 16, 64, 256, 1024, 4096}; optimizers SGD, Adam, Adafactor, MuonValidation loss vs FLOPs (per-FLOP performance), robustness to hyperparameter misspecification, stabilitySmall batch sizes (down to 1) train stably and achieve equal or better per-FLOP loss than larger batches - e.g. GPT-3 1.3B with vanilla SGD at batch size 1 (no momentum, no optimizer state) performs on par with the GPT-3-default AdamW batch-512 baseline, and batch-size-1 Adam is far more robust to LR/beta1/beta2 misspecification than batch 512.Exhaustive grid searches only on 30M model; LR for 1.3B not tuned per optimizer; single-epoch / far-from-convergence regime assumed; results contradict prior findings (vyas2023beyond, filatov2024time).moderatefull-textChallenges the 'small batch is unstable' consensus for LLM pretraining and argues gradient accumulation is wasteful; proposes scaling Adam beta2 by token half-life.
mccandlish2018empiricaltheoretical8 tasks: MNIST, SVHN, CIFAR-10, ImageNet, Billion Word, Atari, Dota 2, and a variational autoencoder on SVHN; SGD with momentum, Adam, RMSPropGradient noise scale B_noise vs critical batch size B_crit (largest useful batch size) and compute/time tradeoffTraining parallelizes almost linearly up to a batch size equal to the gradient noise scale and then switches to a regime of diminishing returns, with measured B_noise predicting B_crit at order-of-magnitude level across all domains (tens of thousands for ImageNet up to millions for Dota 2); the noise scale increases as loss decreases over a training run.Quadratic (Hessian) approximation of loss; assumes well-tuned learning rates; does not address generalization gap; simplified noise scale inaccurate when Hessian is ill-conditioned.highfull-textFoundational critical-batch-size / gradient-noise-scale framework later used to pick batch sizes for GPT-3 and other LLM runs.
merrill2025criticalcase-studyOLMo 1B and 7B (Dolma corpus, 4096-token documents, ~600B-token budget); branched-training measurement of CBS with window Delta=2B tokens, loss tolerance epsilon=0.01; batch-size warmup validation on OLMo 1BCritical batch size B* over training (via local branched training) and final loss with batch-size warmup vs fixed-batch controlsDirectly measured CBS is near 0 at initialization, grows rapidly early in training, then plateaus for OLMo 1B and 7B; the gradient noise scale underestimates the CBS; batch-size warmup trains OLMo 1B to slightly better loss with 43% fewer gradient steps.Measurements on OLMo models (1B, 7B) only; CBS measured in documents, units differ across papers.highfull-textDirect empirical measurement of CBS evolution over LLM pretraining, motivating batch-size warmup schedules to raise data parallelism without loss degradation.
qian2020impacttheoreticalTheory for linear regression and two-layer (extendable to deep) linear networks with Gaussian samples; experiments on MNIST (3-layer FC 1024-512-10), Yelp with XLNet, and other datasets; mini-batch sizes e.g. 32 and 64; SGD and AdamVariance of the stochastic gradient estimator as a function of mini-batch size bThe variance of the SGD gradient estimator is a decreasing function of mini-batch size b (proved for linear regression; a polynomial in 1/b for two-layer linear networks), and empirically both the loss value and the training-test accuracy gap decrease with b, supporting the intuition that smaller batches reach lower loss and generalize better.Exact results only for linear / two-layer linear settings; extension to nonlinear deep networks is empirical and heuristic; link between gradient variance and generalization not proven analytically.moderatefull-textFirst theoretical treatment of gradient variance as a function of mini-batch size, underpinning noise-based explanations of batch-size effects on convergence.
sakip2026copusbenchmarkLLM pretraining workloads with 3B-32B parameter models on 1-4 nodes of 8xH100 and 8xMI210 GPUs; baselines with static parallelism + adaptive or fixed batch size and CBS-only adaptive batch; Megatron-LM/DeepSpeed-style 3D parallelismTime-to-convergence under Goodput (throughput x statistical efficiency) maximizationCo-adaptively tuning global batch size, micro-batch size and 3D parallelism (COPUS) yields average time-to-convergence speedups of 3.9-8.0% over the fastest baseline across four configurations (peak 11.1%), and a fixed Llama-style large-batch recipe (batch 2048 ~ 4M tokens) wastes ~30x wall-clock time early in training (reaches loss 6.2 in 47 min vs under 2 min for COPUS on a 3B/8xH100 run).Small cluster scale (up to 32 GPUs); modest average speedups; arXiv preprint (2026) not peer-reviewed; GNS scaling-factor uncertainty only partially countered by throughput term.moderatefull-textShows global batch size and parallelism strategy choices are coupled, and that batch schedules tracking the evolving critical batch size are needed for wall-clock-efficient LLM training.
shoeybi2021megatronbenchmarkGPT-2-style (up to 8.3B params) and BERT-style (up to 3.9B params) transformers trained on 512 V100 GPUs; GPT-2 models trained with sequence length 1024 at batch size 512 for 300k iterations; WikiText103, LAMBADA, RACEThroughput/scaling efficiency (PetaFLOP/s, % scaling efficiency) and downstream accuracy/perplexityModel+data parallel training sustains 15.1 PetaFLOP/s with 76% scaling efficiency on 512 GPUs for an 8.3B-param GPT-2 trained at 512x1024-token global batch, reaching SOTA WikiText103 perplexity 10.8.Not a batch-size study per se: batch size fixed at 512 and no batch-size ablations; scaling measured in throughput rather than loss-vs-tokens; memory-constrained settings only.highfull-textIndustrial-scale evidence that very large global batches (~0.5M tokens/step) converge to SOTA for multi-billion-parameter LLMs, making large-batch training the norm.
shuai2024scalinglawbenchmarkGPT-series LMs from 125M to 2.6B params trained on up to 300B curated tokens; global batch sizes from 1M to 32M tokens; 3 LR schemes per batch size; extrapolation on 4.3B and 7B models; Huawei Ascend hardwareCross-entropy loss vs compute (FLOPs), tokens, and steps as a function of global batch size and learning rateWith optimal LR, the batch-size scaling laws are Bopt ~ 6.42e3 * C^0.102 for fixed compute and Bopt ~ 3.24e3 * D^0.264 for fixed data (with Bopt ~ D linear when D < ~10B to keep >=~2,500-6,000 steps), implying ~4.7M-token batches for 1T tokens and ~8.7M for 10T tokens; also Nopt ~ 0.297*C^0.464 and Dopt ~ 0.561*C^0.536 (loss fit L = 1.48 + 314.35/N^0.331 + 460.51/D^0.286, R^2=0.962).Law for fixed compute assumed valid for Bopt > 0.5M tokens (C > ~5e18 FLOPs); some conclusions qualitative; single data/tokenizer regime on Ascend hardware.moderatefull-textProvides explicit quantitative batch-size scaling laws for LLM pretraining under compute- or data-constrained budgets, with exponents usable in the review synthesis.
smith2018bayesiantheoreticalTheory (Bayesian evidence + SDE interpretation of SGD) verified on small over-parameterized linear models and small networks; training set sizes N up to ~20k+, learning rates across ~2 orders of magnitudeTest-set accuracy vs batch size; SGD noise scale gInterpreting SGD as a stochastic differential equation gives noise scale g = eps*(N/B - 1) ~ eps*N/B, so with learning rate fixed test accuracy peaks at an optimal batch size B_opt ~ eps*N - verified empirically (B_opt proportional to eps across two orders of magnitude of LR, and proportional to N for N >= ~20000) - explaining the generalization gap where accuracy falls as batch size grows at fixed LR.SDE analysis valid near local minima with stationary covariances; verified on small models only; sharp-minima/Bayesian interpretation debated (e.g. Dinh et al. parameterization critique).moderatefull-textOrigin of the noise-scale view g ~ eps*N/B linking batch size, learning rate and generalization; foundation for later batch-size/LR scaling rules.
smith2018disciplinedbenchmarkResNet-56 on CIFAR-10 (total batch sizes 128/256/512/1024 at near-constant ~26-min execution time), shallow 3-layer net on CIFAR-10, resnet-50 and inception-resnet-v2 on ImageNet; 1cycle LR schedule, cyclical momentum, weight decay sweepsValidation accuracy and loss as a function of learning rate, batch size, momentum, and weight decayWith the 1cycle learning-rate schedule, larger batch sizes are beneficial but with tapering returns - on CIFAR-10/ResNet-56, TBS=512 with proportionally larger LR is the best trade-off (TBS=1024 gives nearly the same accuracy as 512 in fewer iterations), and batch sizes should be compared under constant execution time rather than constant epochs or iterations.Small-scale vision datasets only; heuristic, practical guidance rather than theory; results tied to one hardware setup and the 1cycle schedule.moderatefull-textPractical evidence that batch size, LR and momentum must be co-tuned, and that larger batches require larger learning rates (1cycle) to converge well.
smith2018donotdecaybenchmarkCIFAR-10 and ImageNet with ResNet-50 and Inception-ResNet-V2; SGD, SGD with momentum, Nesterov momentum, and AdamTest accuracy vs number of epochs/parameter updates comparing LR-decay schedules with batch-size-increase schedulesIncreasing the batch size during training reproduces the learning curve of decaying the LR (whenever LR drops by factor alpha, increase batch size by alpha), reaching equivalent test accuracy after the same epochs with fewer parameter updates - ResNet-50 on ImageNet to 76.1% validation accuracy in under 30 minutes, and Inception-ResNet-V2 to 77% in under 2500 updates using batches of 65536 images; scaling B ~ eps further cuts updates, and B ~ 1/(1-m) with momentum slightly reduces accuracy.Vision benchmarks only; method repurposes existing LR schedules rather than deriving new ones; momentum-scaling variant slightly hurts test accuracy; no LLM-scale validation.highfull-textCanonical evidence for batch-size ramp schedules as a substitute for LR decay, enabling large-batch training with fewer optimizer steps.
touvron2023llama2benchmarkLlama 2 7B/13B/34B/70B pretrained on 2T tokens of public data with global batch size 4M tokens (4096 seq len, 1024 batch); AdamW (beta1=0.9, beta2=0.95), cosine LR with 2000-step warmup decaying to 10% of peak, weight decay 0.1; SFT batch 64 at seq len 4096 (LR 2e-5); RLHF PPO batch 512 with mini-batch 64Pretraining loss trajectory and downstream task performance (benchmark suite, human evals)All Llama 2 models were trained with a fixed global batch size of 4M tokens for 2T tokens (peak LR 3e-4 for 7B/13B, 1.5e-4 for 34B/70B), and after pretraining the models still showed no sign of saturation - demonstrating that a fixed large global batch of 4M tokens per step supports converged, SOTA-quality LLM pretraining.No batch-size ablations reported; single fixed batch; industrial report, not a controlled experiment on batch size.highfull-textReal-world production data point: 4M-token global batch per step for 2T-token LLM pretraining, a canonical recipe for the review.
xu2020dynamicallybenchmarkTransformer base (Vaswani et al. setting) on WMT14 English-German and English-French, 2x GTX 1080Ti GPUs; baselines: fixed 25k- and 50k-token batches; dynamic batches via gradient-direction monitoring (alpha=1.1, beta=3)BLEU, training time (100k steps), and statistics of dynamic batch sizesAccumulating mini-batches until the gradient direction starts to fluctuate gives dynamic batch sizes (avg ~26k tokens) that beat the fixed 25k-token baseline by +0.73 (En-De) and +0.82 (En-Fr) BLEU while keeping training time comparable (33h37m vs 35h21m), and alpha=1.0 beats the 25k baseline while being 1.42x faster.NMT tasks only; modest gains; extra monitoring overhead; threshold hyperparameter alpha requires tuning; gradient-direction heuristic lacks theoretical grounding.moderatefull-textEarly empirical evidence that gradient-direction stability can drive adaptive batch-size schedules in Transformer training, improving both convergence quality and efficiency.
you2020largebenchmarkBERT-Large pretraining on TPUv3 Pods; batch sizes from 8K up to 65,536 (with a 131,072 test)time-to-target F1, max batch before optimization stallsLAMB scales BERT-Large pretraining to batches above 32K and cuts wall time from 3 days to 76 minutes; AdamW stops scaling at batch 16K, while LAMB reaches F1 91.345 at 65,536, and going 65,536 to 131,072 gives no further speedup.Optimizer-specific (layer-wise adaptive rates); results on BERT-Large and image classification, not GPT-scale autoregressive models.highfull-textShows the large-batch ceiling is optimizer-dependent for Adam-family methods; layer-wise LR scaling extends it.
zhang2024criticalbenchmarkAutoregressive LMs from 85M to 1.2B params trained on C4 (context length 512, Adam, micro-batch + gradient accumulation); CBS defined as batch size incurring 20% overhead vs linear step scaling; Bopt=256; constant+EWA training beyond fixed durations; plus theory (infinite-width limits, infinite-dimensional least-squares)Critical batch size (steps to target validation loss vs batch size) and its scaling with model size N and data size DIn Chinchilla-optimal runs CBS scales as B* ~ 93.20 * N^0.47 (N in millions), but with data size fixed the model-size law is only B* ~ 621.341 * N^0.087, while scaling data at fixed model size raises CBS as much as joint Chinchilla scaling - CBS scales primarily with data size/training duration, consistent with the theoretical result B*(D) = Theta(D^c) with 0 < c < 1/2 for least-squares mini-batch SGD.Small models (<=1.2B) and single dataset (C4); CBS definition depends on the chosen 20%-overhead convention; power-law fits assume fixed alpha=1 form.highfull-textKey evidence that critical batch size tracks data size rather than model size, informing batch-size schedule design for LLM pretraining.

Swipe sideways to see all columns.

References

  1. Nitish Shirish Keskar et al. (2017). On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — ICLR 2017. Full text read. Origin of the large-batch generalization-gap claim: large batches converge to sharp minima, test-accuracy gap up to 5%.doi:10.48550/arxiv.1609.04836
  2. Priya Goyal et al. (2017). Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — arXiv preprint. Full text read. Linear learning-rate scaling rule plus warmup; ResNet-50 at minibatch 8192 matches batch-256 accuracy.doi:10.48550/arxiv.1706.02677
  3. Samuel L. Smith & Quoc V. Le (2018). A Bayesian Perspective on Generalization and Stochastic Gradient Descent — ICLR 2018. Full text read. SDE/Bayesian view: SGD noise scale g = eps*(N/B-1); optimal batch proportional to eps*N; noise drives SGD to wide minima.doi:10.48550/arxiv.1710.06451
  4. Samuel L. Smith et al. (2018). Don't Decay the Learning Rate, Increase the Batch Size — ICLR 2018. Full text read. Increasing batch size during training reproduces LR decay; B proportional to eps and to 1/(1-m); ImageNet 76.1% in under 30 minutes.doi:10.48550/arxiv.1711.00489
  5. Sam McCandlish et al. (2018). An Empirical Model of Large-Batch Training — NeurIPS 2018. Full text read. Gradient noise scale B_noise predicts the critical batch size across domains; compute-vs-time tradeoff curve.doi:10.48550/arxiv.1812.06162
  6. Joel Hestness et al. (2017). Deep Learning Scaling is Predictable, Empirically — arXiv preprint. Full text read. Early scaling-law work: loss power laws in dataset size; larger models need larger batches and LRs to stay on the trend.doi:10.48550/arxiv.1712.00409
  7. Smith, Leslie N. (2018). A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay — arXiv preprint. Full text read. 1cycle LR schedule and LR range test practice; larger batches help with tapering returns.doi:10.48550/arxiv.1803.09820
  8. Jastrzȩbski, Stanisław et al. (2017). Three Factors Influencing Minima in SGD — arXiv preprint. Full text read. LR-to-batch ratio is the control parameter of SGD dynamics; higher ratio reaches wider minima.doi:10.48550/arxiv.1711.04623
  9. Jared Kaplan et al. (2020). Scaling Laws for Neural Language Models — arXiv preprint. Full text read. Scaling laws: critical batch size is a power law in the loss alone (B* ~ 2e8, alpha_B ~ 0.21); ~1-2M tokens at convergence.doi:10.48550/arxiv.2001.08361
  10. Tom Henighan et al. (2020). Scaling Laws for Autoregressive Generative Modeling — arXiv preprint. Full text read. Cross-domain scaling laws; notes wasted compute if batch size is too large early in training.doi:10.48550/arxiv.2010.14701
  11. Malakar, Sourav (2020). Language Models are Few-Shot Learners — NeurIPS 2020. Full text read. GPT-3: batch ramped linearly from 32K tokens to 0.5M-3.2M (by model size) over the first 4-12B tokens; production anchor.doi:10.48550/arxiv.2005.14165
  12. Chen, Congliang et al. (2022). Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration — Journal of Machine Learning Research. Full text read. Theory: mini-batch and distributed Adam achieve linear speedup in batch size / node count.doi:10.48550/arxiv.2101.05471
  13. Lee, Kiwon et al. (2022). Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions — NeurIPS 2022. Full text read. SGD+momentum saturates at the implicit conditioning ratio: beyond it, convergence matches full-batch momentum.doi:10.48550/arxiv.2206.01029
  14. Hoffmann, Jordan et al. (2022). Training Compute-Optimal Large Language Models — arXiv preprint. Full text read. Chinchilla: compute-optimal N and D scale equally; used a 1.5M to 3M token batch; batch choice per McCandlish heuristics.doi:10.48550/arxiv.2203.15556
  15. Aakanksha Chowdhery et al. (2022). PaLM: Scaling Language Modeling with Pathways — arXiv preprint. Full text read. PaLM: staged batch schedule 1M to 2M to 4M tokens; explicitly trades early sample efficiency for later throughput.doi:10.48550/arxiv.2204.02311
  16. Hugo Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models — arXiv preprint. Full text read. Llama 2: fixed 4M-token global batch for all model sizes across 2T tokens.doi:10.48550/arxiv.2307.09288
  17. Xu, Hongfei et al. (2020). Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change — ACL 2020. Full text read. Dynamic batch size by gradient-direction monitoring: +0.73/+0.82 BLEU over a fixed 25K batch on WMT14.doi:10.18653/v1/2020.acl-main.323
  18. Narayanan, Deepak et al. (2021). Efficient large-scale language model training on GPU clusters using megatron-LM — SC 2021. Full text read. Megatron-LM: 8.3B GPT-2 at 512x1024-token global batches on 512 GPUs; 15.1 PFLOP/s, 76% scaling efficiency.doi:10.1145/3458817.3476209
  19. Li, Shuaipeng et al. (2024). Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling — NeurIPS 2024. Full text read. For Adam, optimal LR vs batch size is non-monotonic ('surge'); peak near the noise scale and moving during training.doi:10.48550/arxiv.2405.14578
  20. Gray, Gavia et al. (2024). Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers — NeurIPS 2024. Full text read. LayerNorm per-example gradients predict full gradient noise scale at zero throughput overhead; GNS-guided batch schedule saves 18% wall time.doi:10.48550/arxiv.2411.00999
  21. Zhang, Hanlin et al. (2024). How Does Critical Batch Size Scale in Pre-training? — arXiv preprint. Full text read. CBS scales with data size, not model size: 93.2*N^0.47 (Chinchilla-optimal) vs N^0.087 with data fixed; 22.91*D^0.47.doi:10.48550/arxiv.2410.21676
  22. Shuai, Xian et al. (2024). Scaling Law for Language Models Training Considering Batch Size — arXiv preprint. Full text read. Batch-aware scaling law: Bopt ~ 6.42e3*C^0.102 (compute frontier) and ~ 3.24e3*D^0.264 (fixed data).doi:10.48550/arxiv.2412.01505
  23. Lau, Tim Tsz-Kit et al. (2024). Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism — arXiv preprint. Full text read. Adaptive (norm-test) batch schedules beat constant and heuristic-warmup baselines; Adam convergence guarantees.doi:10.48550/arxiv.2412.21124
  24. DeepSeek-AI et al. (2024). DeepSeek-V3 Technical Report — arXiv preprint. Full text read. DeepSeek-V3: batch ramped 3,072 to 15,360 sequences (~13M to ~63M tokens at 4K seqlen) over the first 469B tokens.doi:10.48550/arxiv.2412.19437
  25. Aaron Grattafiori et al. (2024). The Llama 3 Herd of Models — arXiv preprint. Full text read. Llama 3 405B: batch doubled 4M to 8M to 16M tokens (seqlen 4096 to 8192); ramp motivated by stability then efficiency.doi:10.48550/arxiv.2407.21783
  26. Merrill, Will et al. (2025). Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training — NeurIPS 2025. Full text read. Direct CBS measurement: near 0 at initialization, grows rapidly, then plateaus; noise-scale proxy unreliable; batch warmup trains OLMo 1B with 43% fewer steps.doi:10.48550/arxiv.2505.23971
  27. Bergsma, Shane et al. (2025). Power Lines: Scaling laws for weight decay and batch size in LLM pre-training — NeurIPS 2025. Full text read. B_opt and B_crit follow power laws in dataset tokens D, independent of N (B_crit ~ 0.0471*D^0.47); AdamW timescale tau = B/(eta*lambda*D).doi:10.48550/arxiv.2505.13738
  28. Marek, Martin et al. (2025). Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful — NeurIPS 2025. Full text read. Batch-1 vanilla SGD matches tuned AdamW at equal tokens; small batches more robust to optimizer and hyperparameter choice.doi:10.48550/arxiv.2507.07101
  29. Kutuzov, Nikolay et al. (2025). AdLoCo: adaptive batching significantly improves communications efficiency and convergence for Large Language Models — arXiv preprint. Full text read. Adaptive local batch sizes for low-communication (DiLoCo-style) distributed training.doi:10.48550/arxiv.2508.18182
  30. Qian, X. & Klabjan, Diego (2020). The Impact of the Mini-batch Size on the Variance of Gradients in Stochastic Gradient Descent — arXiv preprint. Full text read. Theory: variance of the SGD gradient estimator decreases with mini-batch size.doi:10.48550/arxiv.2004.13146
  31. Sakip, Akhmed et al. (2026). COPUS: Co-adaptive Parallelism and Batch Size Selection in Large Language Model Training — arXiv preprint. Full text read. Co-adapting batch size and 3D parallelism online: 3.9-8.0% time-to-convergence speedups (peak 11.1%).doi:10.48550/arxiv.2604.26687
  32. You, Yang et al. (2020). Large Batch Optimization for Deep Learning: Training BERT in 76 minutes — ICLR 2020. Full text read. LAMB layer-wise adaptive rates let Adam-family training scale past 32K batch; BERT-Large in 76 minutes, AdamW stalls at 16K.doi:10.48550/arxiv.1904.00962
  33. Geiping, Jonas & Goldstein, Tom (2023). Cramming: Training a Language Model on a Single GPU in One Day — ICML 2023. Full text read. Single-GPU BERT cramming: micro-batch 96 is far below the optimal batch (1536 min loss / 4032 best downstream).doi:10.48550/arxiv.2212.14034