Literature Reviews
Long-form, source-checked answers to research questions.
Rapid ≈ a weekend read · Standard ≈ 40 sources · Deep ≈ 80+ sources
Filter
Distributed low-communication training with DiLoCo and checkpointless training
How do DiLoCo-style distributed low-communication training and checkpointless training work, what do they deliver, and how do they combine?
Two research lines now attack the two biggest taxes on very large model training: the synchronization tax (DiLoCo-style methods cut communication by 100-500x so training can span clusters, continents, even the public internet) and the checkpoint tax (recovery without full checkpoint-restart, shipped first by AWS as 'checkpointless training'). The evidence shows both work, they combine naturally, and the deepest synthesis yet - Decoupled DiLoCo - trains through continuous hardware failures with zero global downtime. Caveat: 'checkpointless' is a product term, not an academic one, and the most-cited 10B cross-continent result circulates under a name ('DiLoCo-XL') no primary source uses.
Smoothing the power curve of LLM training and inference
Which kernel- and scheduler-level techniques reduce power ripples during LLM training and inference, and at what cost in energy and throughput?
GPU power draw during LLM training and inference is bursty: compute-bound prefill spikes, memory-bound decode idles far below the power ceiling, and synchronised training steps add further lumps. Across 89 sources (2011-2026) the evidence shows GPU frequency and power-capping settings have pronounced energy sweet spots (routinely 15-35% energy saved for single-digit throughput loss) that apply only to compute-bound phases; the newest work shows the power ramp between scheduling steps, not the peak, is the controllable quantity. Confidence is moderate: device-level trade-offs are replicated across many groups, while the ripple-smoothing literature is young (mostly 2025-2026 preprints) and largely single-node or emulation-scale.
Synthetic workload generation for benchmarking LLM inference
Which open-source tools and datasets can generate synthetic workloads — chat, agentic, and other — for benchmarking LLM inference performance?
A practical map of the tools that generate workloads for benchmarking LLM inference: engine-native load generators (vLLM, SGLang, llama.cpp), trace-driven generators (BurstGPT, Azure traces, ShareGPT/LMSYS replay), statistical generators calibrated to production traffic (ServeGen, FineServe), simulators (LLMServingSim, Frontier, SimLLM), and the emerging agentic-workload tooling (XPerf, SGLang agentic-trace, GuideLLM tool calling). The evidence says workload realism matters — Poisson arrivals and fixed token lengths misrepresent production traffic — and that agentic workloads are the least standardized but fastest-moving category. Confidence is moderate: repo capabilities were verified from primary docs fetched this session, but several tools change weekly and one major tool (NVIDIA GenAI-Perf) was unretrievable.
Selling infrastructure to inference providers: what a neocloud can offer without competing
Which inference providers should an AI-factory builder target, at which infrastructure layer, and what must it know about inference to sell them capacity without becoming their competitor?
The inference-provider market splits into open-weight GPU hosts (Together, Fireworks, DeepInfra, Baseten, Modal — they rent nearly all their capacity and compete on serving software) and closed-weight labs (OpenAI, Anthropic, Google, Meta, xAI — they rent at enormous scale via take-or-pay contracts but vertically integrate software and increasingly silicon). The evidence says a neocloud should target the open-weight hosts at the fleet/facility and control-plane layers — power, cooling, density, grid, orchestration that respects the customer's serving engine — and never compete at the serving-engine layer, which is the customer's moat. Firmus's public record (AI FactoryOS, HyperCube, Model-to-Grid, a Fireworks partnership) already points this way. Confidence is moderate: the market facts are industry-reported, and several claimed product names could not be verified in any public source.
The LLM compute market — training, fine-tuning, and inference
How is LLM compute demand, revenue, and profit split between training, fine-tuning, and inference, and what does the shift toward inference mean for AI factory builders?
Companies spent $37B on enterprise generative AI in 2025 and Gartner counts $2.59T of total AI spending for 2026, and the mix is shifting decisively from training to inference — inference overtakes training in AI-optimized cloud spend by 2026 and across the wider market by 2029. Fine-tuning is a real but small slice of that spend, agentic workloads are the fastest-growing inference category, and frontier training is multimodal-first even though public accounting of compute-by-modality barely exists. The evidence says an AI factory builder should not sell inference tokens against its own customers, but should become inference-workload-aware as infrastructure — disaggregated prefill and decode, SLO-aware scheduling, and tokens-per-watt are all sellable attributes.
The LLM inference engine playbook: tricks in vLLM and SGLang
Which optimization techniques do the vLLM and SGLang serving engines implement, and in what order should an engineer study them?
LLM serving engines are built from a stack of optimization tricks, each attacking one bottleneck: batching keeps the GPU busy, KV-cache paging and reuse stop memory from fragmenting or being recomputed, faster attention kernels and lower-precision arithmetic cut bytes moved, speculative decoding trades spare compute for fewer serial steps, and prefill-decode disaggregation lets two workloads with opposite resource profiles run on machines tuned for each. This review catalogs the trick list of the two dominant open-source engines, vLLM and SGLang, maps each trick to the paper that introduced it and the bottleneck it addresses, and closes with an ordered self-study path. Confidence is moderate: the mechanisms are well established in the literature, but the engines' own performance claims are vendor-reported and rarely independently benchmarked.
Benchmarking tokens per watt: how AI inference energy efficiency is measured
What benchmarks and measurement methodologies exist for quantifying AI inference energy efficiency in tokens per watt (or per joule), from accelerator to data-centre level, and what do they establish?
The literature on measuring AI inference efficiency in tokens-per-watt terms is young (mostly 2023-2026) and fragmented: one consortium standard exists at the system level (MLPerf Power), but the benchmarks that actually report tokens per watt or joules per token are research tools with incompatible measurement boundaries. Measured numbers span orders of magnitude — roughly 3-4 joules per output token for a 65B model on A100s, 0.002 to 2.9 kWh per 1,000 inferences depending on task, a 65x spread across models in commercial data centres, and a proposed 1/W law under which tokens per watt halves each time the context window doubles. No retrieved benchmark measures tokens per watt at the data-centre (facility) level; facility efficiency is still expressed as PUE, so wall-level tokens per watt is derived, not measured. Confidence is moderate: 20 of 65 sources were read in full text and several prominent items were unreachable in-session.
Kernel optimization for LLM training: techniques, trade-offs, and use cases
Which kernel-level optimization techniques most improve LLM training efficiency, and what are the trade-offs and use cases of each?
Kernel-level optimization is where most LLM training efficiency gains of the last six years come from, and the field splits into five families: fused attention kernels, low-precision GEMM kernels, mixture-of-experts kernels, communication-overlap kernels, and fused elementwise/optimizer kernels. The evidence is consistent that each family delivers large, workload-specific gains — 2x on attention, 25-75% on FP8 training, 1.4-5.7x on MoE layers, 1.2-2.9x on long-context training — but the gains are conditional on hardware generation, model architecture, and engineering effort, and almost every headline number is vendor- or single-group-benchmarked. The main caveat: no shared benchmark exists, so cross-paper 'x-times faster' claims are not directly comparable.
GPU kernel development for LLM inference: Triton, TileLang, FlashMLA, and FlashInfer compared
How do Triton, TileLang, FlashMLA, and FlashInfer compare as GPU kernel development tools for LLM inference, and which use cases does each serve best?
The four tools fall into two families: kernel DSLs and compilers (Triton, TileLang) that generate kernels from Python-like tile programs, and attention kernel libraries and engines (FlashInfer, FlashMLA) that ship pre-built or JIT-compiled CUDA kernels for serving. The evidence shows DSLs are within reach of hand-tuned CUDA on GEMM and attention once tuned (59-105% of vendor baselines depending on workload and effort), while the libraries deliver 29-69% lower inter-token latency than compiler backends in serving benchmarks and dominate production deployment. FlashMLA is uniquely tied to one workload, DeepSeek's MLA, and has no peer-reviewed paper, so it is documented here through the DeepSeek technical reports and third-party kernel evaluations. On the training side the same DSLs power the ecosystem layer (torch.compile, Liger, MOSS's Triton FP8 GEMMs), but the training-kernel frontier — block-sparse MoE, single-kernel distributed MoE, FP8 training — is written in expert CUDA and CUTLASS, and FlashInfer and FlashMLA remain forward-only serving libraries. Confidence is moderate: almost all performance claims come from vendor-authored or single-group benchmarks with no independent head-to-head of all four tools.
Global batch size and convergence in large language model training
How does global batch size affect the convergence and compute efficiency of LLM pretraining?
The global batch size (tokens per optimizer step) does not change where training converges so much as how efficiently it gets there: it trades steps against tokens against wall-clock. A measurable quantity, the gradient noise scale, marks the critical batch size beyond which larger batches buy almost no extra speed, and the evidence from both theory and production LLM runs says the critical batch size grows with training progress and with dataset size, not model size. Learning-rate scaling rules (linear for SGD, subtler for Adam) are what make large batches safe, and batch ramping is now standard practice across GPT-3, PaLM, Llama 3, and DeepSeek-V3.
Recursive self-improvement in large language models
How do large language models improve themselves recursively, through which mechanisms, with what empirical results, and under what limits?
This review asks how large language models improve themselves recursively. Across 156 sources spanning 2006 to 2026, single-loop self-improvement — self-training, self-correction, self-play, self-rewarding — is repeatedly shown to deliver real gains, but every mechanism also has a documented failure mode, and the recursion reliably stalls or degrades without an external anchor such as verifiable feedback, fresh data, or human judgment. Confidence is moderate: the field is young, dominated by author-run benchmarks and preprints, and several headline results are actively contested.
ECC memory errors: rates, causes, and consequences in datacenter and AI systems
What does the evidence show about the rate, causes, and system-level consequences of ECC-protected memory errors in modern computing systems, and how effective are the mitigations?
Error-correcting-code (ECC) memory protects datacenter and AI systems from bit flips that would otherwise corrupt results, but the evidence shows the protection is partial and the threat is changing. Field studies spanning 1979-2026 find real DRAM error rates orders of magnitude above lab estimates (more than 8% of DIMMs affected per year at Google), show that most errors are hard, repeatable faults rather than cosmic-ray soft errors, and document that ECC catches most but not all of them - single-bit-correction codes can miscorrect double-bit errors, on-die ECC hides raw error patterns from operators, and GPU/HBM studies reveal error rates varying by three orders of magnitude across otherwise identical clusters. The largest caveat: the load-bearing field studies are from a handful of large operators (Google, Meta, LANL, BSC, Alibaba), and AI-specific evidence is young, with several key GPU/HBM results still preprints.
Kernel fusion for efficient LLM training and inference
Which kernel-fusion methods improve the efficiency of large language model training and inference, and what gains do they deliver?
Kernel fusion — computing several operations in one GPU kernel to cut memory traffic and launch overhead — is the dominant mechanism behind the efficiency gains of the past six years. The evidence is consistent that attention kernels (FlashAttention and its descendants) deliver 2–4x speedups and near-constant memory in sequence length, that fused training-op kernels (Liger and peers) recover roughly 20% throughput and 60% memory on the training side, and that FP8 pipelines now train 10–75% faster with matched quality. Most evidence is preprint-grade and hardware-specific; the field lacks a shared benchmark, and the newest frontier is using LLMs themselves to write the kernels.
Optimizing large language model inference
Which techniques most improve the throughput, latency, and cost of large language model inference?
Large language model inference is memory-bound, not compute-bound, and every major optimization family attacks the memory wall from a different angle: batching and scheduling to keep GPUs busy, KV-cache management to shrink the state that grows with context, quantization to cut bytes per weight, speculative decoding to trade spare compute for fewer serial steps, and prefill-decode separation to stop two workloads with opposite resource profiles from interfering. The evidence is consistent that these techniques deliver large gains — commonly 2–6x in throughput or latency, occasionally 10x or more in narrow regimes — but nearly every headline number is an "up to" figure measured on the authors' own hardware and workload, and independent benchmarks are only beginning to appear.
Interconnects for large-scale LLM training and inference: RDMA, InfiniBand, NVLink, and the network stack
What does the evidence from 2020 to 2026 show about how interconnect technologies (RDMA, InfiniBand, NVLink, Ethernet/RoCE, and the software stacks built on them) shape the performance, scaling, and cost of large-scale LLM training and inference?
Large-scale LLM training and inference now depend on a hierarchy of interconnects: NVLink and NVSwitch inside servers, InfiniBand or RoCE Ethernet between them, and collective-communication libraries that turn raw fabric into training progress. The 2020-2026 evidence shows the lossless RDMA fabric that carried early GPT-era training is under strain, with congestion-control pathologies and head-of-line blocking documented in both directions, while production operators engineer around it and a lossy, packet-spraying Ultra Ethernet path is emerging. Meanwhile Mixture-of-Experts models and long-context inference have made all-to-all traffic and KV-cache movement the new communication bottlenecks, and NVLink scale-up domains, UALink, and CXL memory pooling are reshaping where computation and memory live. Confidence is moderate: the transport and collective stories are replicated across many independent studies, but the newest hardware (NVLink 4/5, UALink, GB200-class superpods) is covered mainly by vendor-adjacent or preprint evidence.
Training full-duplex LLMs for spoken dialogue
How are full-duplex (simultaneous bidirectional speech) LLMs trained, and which architectures, data, and training methods achieve low-latency, interruptible spoken dialogue?
Full-duplex spoken dialogue models — systems that listen and speak at the same time — became a distinct research field between 2023 and 2026, moving from the first speech-in/speech-out language models to a crowded design space of dual-stream transformers, frozen-LLM adapters, and single-stream 'native duplex' recipes. The evidence shows training data that contains genuine overlapping speech is the binding constraint, not architecture; latency and interruption handling are now measurable but benchmarks disagree about what to measure; and post-training alignment (RL/DPO on turn-taking behavior) is the fastest-moving lever. Confidence is moderate: the field is young, most frontier systems are preprints, and no standardized evaluation exists yet.
Optimizing MLPerf Training closed-division submissions: what the literature shows
Which methods documented in the literature most reduce time-to-quality for MLPerf Training closed-division submissions, given the rule that model, optimizer and most hyperparameters are fixed?
A review of 47 sources on how to minimize time-to-quality in the MLPerf Training closed division, where the model, optimizer, and most hyperparameters are fixed by the rules. The levers the evidence supports are: picking the global batch size that sits at the reference-convergence boundary, reduced-precision training (bf16/fp8, now also fp4), distributed-parallelism and communication-overlap engineering, data-pipeline and kernel work, and benchmark-specific plays for DLRM, LoRA fine-tuning, and MoE pretraining. Each lever is worth roughly 1.2–2× in isolation and they compound, but the convergence-speed cap (Reference Convergence Points) means the winning move is maximum throughput at the RCP boundary, not faster convergence. Confidence is moderate: the per-lever results are replicated systems literature, but no public study directly tests the full closed-division strategy.
Energy efficiency across the AI datacenter stack
What does the 2023-2026 literature establish about improving energy efficiency across the AI datacenter stack - from GPU power management and workload scheduling to grid-level coordination - and how strong is the evidence for each lever?
AI datacenters are among the fastest-growing electricity consumers of the 2020s, and the 2023-2026 literature offers a stack of levers to improve their efficiency: GPU power capping and frequency control, energy- and carbon-aware scheduling, LLM serving configuration, cooling control, and grid-interactive operation. Evidence is strongest at the device and workload levels (measured savings of roughly 15-50% on specific workloads), moderate for scheduling (mostly simulation), and thinnest at grid level, where a single production deployment carries much of the weight. The key caveat: nearly every lever is workload- and hardware-generation-specific - power capping that saves ~23% energy on one model hurts SLOs on another - so none generalises without measurement.
[Sample] Durability of protection from mRNA COVID-19 vaccines
How durable is the protection conferred by two-dose mRNA COVID-19 vaccines, and which immune measures predict waning?
Two-dose mRNA vaccines show high short-term efficacy, but protection against infection wanes within months while protection against severe disease holds longer. Neutralising antibody titres track that waning and are the best available correlate. This is the seeded sample review that exercises the site's templates end to end.
No reviews match these filters.