https://reviews.lewiswon.me Auditable literature reviews: every claim tied to a verified source. Sun, 06 Sep 2026 14:27:06 GMT https://reviews.lewiswon.me/reviews/distributed-diloco-checkpointless-training/ https://reviews.lewiswon.me/reviews/distributed-diloco-checkpointless-training/ Sun, 06 Sep 2026 00:00:00 GMT Two research lines now attack the two biggest taxes on very large model training: the synchronization tax (DiLoCo-style methods cut communication by 100-500x so training can span clusters, continents, even the public internet) and the checkpoint tax (recovery without full checkpoint-restart, shipped first by AWS as 'checkpointless training'). The evidence shows both work, they combine naturally, and the deepest synthesis yet - Decoupled DiLoCo - trains through continuous hardware failures with zero global downtime. Caveat: 'checkpointless' is a product term, not an academic one, and the most-cited 10B cross-continent result circulates under a name ('DiLoCo-XL') no primary source uses. distributed systems, machine learning https://reviews.lewiswon.me/reviews/llm-gpu-power-shaping/ https://reviews.lewiswon.me/reviews/llm-gpu-power-shaping/ Sun, 06 Sep 2026 00:00:00 GMT GPU power draw during LLM training and inference is bursty: compute-bound prefill spikes, memory-bound decode idles far below the power ceiling, and synchronised training steps add further lumps. Across 89 sources (2011-2026) the evidence shows GPU frequency and power-capping settings have pronounced energy sweet spots (routinely 15-35% energy saved for single-digit throughput loss) that apply only to compute-bound phases; the newest work shows the power ramp between scheduling steps, not the peak, is the controllable quantity. Confidence is moderate: device-level trade-offs are replicated across many groups, while the ripple-smoothing literature is young (mostly 2025-2026 preprints) and largely single-node or emulation-scale. computer systems, energy systems, high-performance computing https://reviews.lewiswon.me/reviews/synthetic-workload-llm-inference/ https://reviews.lewiswon.me/reviews/synthetic-workload-llm-inference/ Tue, 01 Sep 2026 00:00:00 GMT A practical map of the tools that generate workloads for benchmarking LLM inference: engine-native load generators (vLLM, SGLang, llama.cpp), trace-driven generators (BurstGPT, Azure traces, ShareGPT/LMSYS replay), statistical generators calibrated to production traffic (ServeGen, FineServe), simulators (LLMServingSim, Frontier, SimLLM), and the emerging agentic-workload tooling (XPerf, SGLang agentic-trace, GuideLLM tool calling). The evidence says workload realism matters — Poisson arrivals and fixed token lengths misrepresent production traffic — and that agentic workloads are the least standardized but fastest-moving category. Confidence is moderate: repo capabilities were verified from primary docs fetched this session, but several tools change weekly and one major tool (NVIDIA GenAI-Perf) was unretrievable. computer systems, machine learning systems https://reviews.lewiswon.me/reviews/inference-neocloud-strategy/ https://reviews.lewiswon.me/reviews/inference-neocloud-strategy/ Sun, 23 Aug 2026 00:00:00 GMT The inference-provider market splits into open-weight GPU hosts (Together, Fireworks, DeepInfra, Baseten, Modal — they rent nearly all their capacity and compete on serving software) and closed-weight labs (OpenAI, Anthropic, Google, Meta, xAI — they rent at enormous scale via take-or-pay contracts but vertically integrate software and increasingly silicon). The evidence says a neocloud should target the open-weight hosts at the fleet/facility and control-plane layers — power, cooling, density, grid, orchestration that respects the customer's serving engine — and never compete at the serving-engine layer, which is the customer's moat. Firmus's public record (AI FactoryOS, HyperCube, Model-to-Grid, a Fireworks partnership) already points this way. Confidence is moderate: the market facts are industry-reported, and several claimed product names could not be verified in any public source. industry economics, AI infrastructure https://reviews.lewiswon.me/reviews/llm-compute-market-economics/ https://reviews.lewiswon.me/reviews/llm-compute-market-economics/ Wed, 19 Aug 2026 00:00:00 GMT Companies spent $37B on enterprise generative AI in 2025 and Gartner counts $2.59T of total AI spending for 2026, and the mix is shifting decisively from training to inference — inference overtakes training in AI-optimized cloud spend by 2026 and across the wider market by 2029. Fine-tuning is a real but small slice of that spend, agentic workloads are the fastest-growing inference category, and frontier training is multimodal-first even though public accounting of compute-by-modality barely exists. The evidence says an AI factory builder should not sell inference tokens against its own customers, but should become inference-workload-aware as infrastructure — disaggregated prefill and decode, SLO-aware scheduling, and tokens-per-watt are all sellable attributes. industry economics, AI infrastructure https://reviews.lewiswon.me/reviews/llm-inference-engine-tricks/ https://reviews.lewiswon.me/reviews/llm-inference-engine-tricks/ Tue, 18 Aug 2026 00:00:00 GMT LLM serving engines are built from a stack of optimization tricks, each attacking one bottleneck: batching keeps the GPU busy, KV-cache paging and reuse stop memory from fragmenting or being recomputed, faster attention kernels and lower-precision arithmetic cut bytes moved, speculative decoding trades spare compute for fewer serial steps, and prefill-decode disaggregation lets two workloads with opposite resource profiles run on machines tuned for each. This review catalogs the trick list of the two dominant open-source engines, vLLM and SGLang, maps each trick to the paper that introduced it and the bottleneck it addresses, and closes with an ordered self-study path. Confidence is moderate: the mechanisms are well established in the literature, but the engines' own performance claims are vendor-reported and rarely independently benchmarked. computer science, machine learning systems https://reviews.lewiswon.me/reviews/tokens-per-watt-benchmarks/ https://reviews.lewiswon.me/reviews/tokens-per-watt-benchmarks/ Mon, 17 Aug 2026 00:00:00 GMT The literature on measuring AI inference efficiency in tokens-per-watt terms is young (mostly 2023-2026) and fragmented: one consortium standard exists at the system level (MLPerf Power), but the benchmarks that actually report tokens per watt or joules per token are research tools with incompatible measurement boundaries. Measured numbers span orders of magnitude — roughly 3-4 joules per output token for a 65B model on A100s, 0.002 to 2.9 kWh per 1,000 inferences depending on task, a 65x spread across models in commercial data centres, and a proposed 1/W law under which tokens per watt halves each time the context window doubles. No retrieved benchmark measures tokens per watt at the data-centre (facility) level; facility efficiency is still expressed as PUE, so wall-level tokens per watt is derived, not measured. Confidence is moderate: 20 of 65 sources were read in full text and several prominent items were unreachable in-session. computer systems, energy systems, machine learning https://reviews.lewiswon.me/reviews/kernel-optimization-llm-training/ https://reviews.lewiswon.me/reviews/kernel-optimization-llm-training/ Sat, 15 Aug 2026 00:00:00 GMT Kernel-level optimization is where most LLM training efficiency gains of the last six years come from, and the field splits into five families: fused attention kernels, low-precision GEMM kernels, mixture-of-experts kernels, communication-overlap kernels, and fused elementwise/optimizer kernels. The evidence is consistent that each family delivers large, workload-specific gains — 2x on attention, 25-75% on FP8 training, 1.4-5.7x on MoE layers, 1.2-2.9x on long-context training — but the gains are conditional on hardware generation, model architecture, and engineering effort, and almost every headline number is vendor- or single-group-benchmarked. The main caveat: no shared benchmark exists, so cross-paper 'x-times faster' claims are not directly comparable. computer science, machine learning systems, high-performance computing https://reviews.lewiswon.me/reviews/gpu-kernel-tools-llm-inference/ https://reviews.lewiswon.me/reviews/gpu-kernel-tools-llm-inference/ Thu, 13 Aug 2026 00:00:00 GMT The four tools fall into two families: kernel DSLs and compilers (Triton, TileLang) that generate kernels from Python-like tile programs, and attention kernel libraries and engines (FlashInfer, FlashMLA) that ship pre-built or JIT-compiled CUDA kernels for serving. The evidence shows DSLs are within reach of hand-tuned CUDA on GEMM and attention once tuned (59-105% of vendor baselines depending on workload and effort), while the libraries deliver 29-69% lower inter-token latency than compiler backends in serving benchmarks and dominate production deployment. FlashMLA is uniquely tied to one workload, DeepSeek's MLA, and has no peer-reviewed paper, so it is documented here through the DeepSeek technical reports and third-party kernel evaluations. On the training side the same DSLs power the ecosystem layer (torch.compile, Liger, MOSS's Triton FP8 GEMMs), but the training-kernel frontier — block-sparse MoE, single-kernel distributed MoE, FP8 training — is written in expert CUDA and CUTLASS, and FlashInfer and FlashMLA remain forward-only serving libraries. Confidence is moderate: almost all performance claims come from vendor-authored or single-group benchmarks with no independent head-to-head of all four tools. computer science, machine learning systems, high-performance computing https://reviews.lewiswon.me/reviews/batch-size-convergence-llm/ https://reviews.lewiswon.me/reviews/batch-size-convergence-llm/ Mon, 10 Aug 2026 00:00:00 GMT The global batch size (tokens per optimizer step) does not change where training converges so much as how efficiently it gets there: it trades steps against tokens against wall-clock. A measurable quantity, the gradient noise scale, marks the critical batch size beyond which larger batches buy almost no extra speed, and the evidence from both theory and production LLM runs says the critical batch size grows with training progress and with dataset size, not model size. Learning-rate scaling rules (linear for SGD, subtler for Adam) are what make large batches safe, and batch ramping is now standard practice across GPT-3, PaLM, Llama 3, and DeepSeek-V3. machine learning, distributed systems https://reviews.lewiswon.me/reviews/recursive-self-improvement-llms/ https://reviews.lewiswon.me/reviews/recursive-self-improvement-llms/ Mon, 10 Aug 2026 00:00:00 GMT This review asks how large language models improve themselves recursively. Across 156 sources spanning 2006 to 2026, single-loop self-improvement — self-training, self-correction, self-play, self-rewarding — is repeatedly shown to deliver real gains, but every mechanism also has a documented failure mode, and the recursion reliably stalls or degrades without an external anchor such as verifiable feedback, fresh data, or human judgment. Confidence is moderate: the field is young, dominated by author-run benchmarks and preprints, and several headline results are actively contested. computer science, machine learning https://reviews.lewiswon.me/reviews/ecc-memory-errors/ https://reviews.lewiswon.me/reviews/ecc-memory-errors/ Sat, 08 Aug 2026 00:00:00 GMT Error-correcting-code (ECC) memory protects datacenter and AI systems from bit flips that would otherwise corrupt results, but the evidence shows the protection is partial and the threat is changing. Field studies spanning 1979-2026 find real DRAM error rates orders of magnitude above lab estimates (more than 8% of DIMMs affected per year at Google), show that most errors are hard, repeatable faults rather than cosmic-ray soft errors, and document that ECC catches most but not all of them - single-bit-correction codes can miscorrect double-bit errors, on-die ECC hides raw error patterns from operators, and GPU/HBM studies reveal error rates varying by three orders of magnitude across otherwise identical clusters. The largest caveat: the load-bearing field studies are from a handful of large operators (Google, Meta, LANL, BSC, Alibaba), and AI-specific evidence is young, with several key GPU/HBM results still preprints. computer systems, computer architecture, reliability engineering https://reviews.lewiswon.me/reviews/kernel-fusion-llm-efficiency/ https://reviews.lewiswon.me/reviews/kernel-fusion-llm-efficiency/ Sat, 08 Aug 2026 00:00:00 GMT Kernel fusion — computing several operations in one GPU kernel to cut memory traffic and launch overhead — is the dominant mechanism behind the efficiency gains of the past six years. The evidence is consistent that attention kernels (FlashAttention and its descendants) deliver 2–4x speedups and near-constant memory in sequence length, that fused training-op kernels (Liger and peers) recover roughly 20% throughput and 60% memory on the training side, and that FP8 pipelines now train 10–75% faster with matched quality. Most evidence is preprint-grade and hardware-specific; the field lacks a shared benchmark, and the newest frontier is using LLMs themselves to write the kernels. computer science, machine learning systems, high-performance computing https://reviews.lewiswon.me/reviews/llm-inference-optimization/ https://reviews.lewiswon.me/reviews/llm-inference-optimization/ Sat, 08 Aug 2026 00:00:00 GMT Large language model inference is memory-bound, not compute-bound, and every major optimization family attacks the memory wall from a different angle: batching and scheduling to keep GPUs busy, KV-cache management to shrink the state that grows with context, quantization to cut bytes per weight, speculative decoding to trade spare compute for fewer serial steps, and prefill-decode separation to stop two workloads with opposite resource profiles from interfering. The evidence is consistent that these techniques deliver large gains — commonly 2–6x in throughput or latency, occasionally 10x or more in narrow regimes — but nearly every headline number is an "up to" figure measured on the authors' own hardware and workload, and independent benchmarks are only beginning to appear. computer science, machine learning systems https://reviews.lewiswon.me/reviews/llm-interconnects/ https://reviews.lewiswon.me/reviews/llm-interconnects/ Sat, 08 Aug 2026 00:00:00 GMT Large-scale LLM training and inference now depend on a hierarchy of interconnects: NVLink and NVSwitch inside servers, InfiniBand or RoCE Ethernet between them, and collective-communication libraries that turn raw fabric into training progress. The 2020-2026 evidence shows the lossless RDMA fabric that carried early GPT-era training is under strain, with congestion-control pathologies and head-of-line blocking documented in both directions, while production operators engineer around it and a lossy, packet-spraying Ultra Ethernet path is emerging. Meanwhile Mixture-of-Experts models and long-context inference have made all-to-all traffic and KV-cache movement the new communication bottlenecks, and NVLink scale-up domains, UALink, and CXL memory pooling are reshaping where computation and memory live. Confidence is moderate: the transport and collective stories are replicated across many independent studies, but the newest hardware (NVLink 4/5, UALink, GB200-class superpods) is covered mainly by vendor-adjacent or preprint evidence. computer systems, computer networks, computer architecture https://reviews.lewiswon.me/reviews/full-duplex-llm-training/ https://reviews.lewiswon.me/reviews/full-duplex-llm-training/ Fri, 07 Aug 2026 00:00:00 GMT Full-duplex spoken dialogue models — systems that listen and speak at the same time — became a distinct research field between 2023 and 2026, moving from the first speech-in/speech-out language models to a crowded design space of dual-stream transformers, frozen-LLM adapters, and single-stream 'native duplex' recipes. The evidence shows training data that contains genuine overlapping speech is the binding constraint, not architecture; latency and interruption handling are now measurable but benchmarks disagree about what to measure; and post-training alignment (RL/DPO on turn-taking behavior) is the fastest-moving lever. Confidence is moderate: the field is young, most frontier systems are preprints, and no standardized evaluation exists yet. machine learning, speech processing, natural language processing https://reviews.lewiswon.me/reviews/mlperf-training-closed-optimization/ https://reviews.lewiswon.me/reviews/mlperf-training-closed-optimization/ Fri, 07 Aug 2026 00:00:00 GMT A review of 47 sources on how to minimize time-to-quality in the MLPerf Training closed division, where the model, optimizer, and most hyperparameters are fixed by the rules. The levers the evidence supports are: picking the global batch size that sits at the reference-convergence boundary, reduced-precision training (bf16/fp8, now also fp4), distributed-parallelism and communication-overlap engineering, data-pipeline and kernel work, and benchmark-specific plays for DLRM, LoRA fine-tuning, and MoE pretraining. Each lever is worth roughly 1.2–2× in isolation and they compound, but the convergence-speed cap (Reference Convergence Points) means the winning move is maximum throughput at the RCP boundary, not faster convergence. Confidence is moderate: the per-lever results are replicated systems literature, but no public study directly tests the full closed-division strategy. computer systems, machine learning https://reviews.lewiswon.me/reviews/model-to-grid-ai-datacenter-efficiency/ https://reviews.lewiswon.me/reviews/model-to-grid-ai-datacenter-efficiency/ Fri, 07 Aug 2026 00:00:00 GMT AI datacenters are among the fastest-growing electricity consumers of the 2020s, and the 2023-2026 literature offers a stack of levers to improve their efficiency: GPU power capping and frequency control, energy- and carbon-aware scheduling, LLM serving configuration, cooling control, and grid-interactive operation. Evidence is strongest at the device and workload levels (measured savings of roughly 15-50% on specific workloads), moderate for scheduling (mostly simulation), and thinnest at grid level, where a single production deployment carries much of the weight. The key caveat: nearly every lever is workload- and hardware-generation-specific - power capping that saves ~23% energy on one model hurts SLOs on another - so none generalises without measurement. computer systems, high-performance computing, energy systems https://reviews.lewiswon.me/reviews/mrna-vaccine-durability/ https://reviews.lewiswon.me/reviews/mrna-vaccine-durability/ Fri, 07 Aug 2026 00:00:00 GMT Two-dose mRNA vaccines show high short-term efficacy, but protection against infection wanes within months while protection against severe disease holds longer. Neutralising antibody titres track that waning and are the best available correlate. This is the seeded sample review that exercises the site's templates end to end. vaccinology, infectious disease epidemiology