Tag: Mixture-of-Experts
3 reviews
Kernel optimization for LLM training: techniques, trade-offs, and use cases
Which kernel-level optimization techniques most improve LLM training efficiency, and what are the trade-offs and use cases of each?
Kernel-level optimization is where most LLM training efficiency gains of the last six years come from, and the field splits into five families: fused attention kernels, low-precision GEMM kernels, mixture-of-experts kernels, communication-overlap kernels, and fused elementwise/optimizer kernels. The evidence is consistent that each family delivers large, workload-specific gains — 2x on attention, 25-75% on FP8 training, 1.4-5.7x on MoE layers, 1.2-2.9x on long-context training — but the gains are conditional on hardware generation, model architecture, and engineering effort, and almost every headline number is vendor- or single-group-benchmarked. The main caveat: no shared benchmark exists, so cross-paper 'x-times faster' claims are not directly comparable.
Kernel fusion for efficient LLM training and inference
Which kernel-fusion methods improve the efficiency of large language model training and inference, and what gains do they deliver?
Kernel fusion — computing several operations in one GPU kernel to cut memory traffic and launch overhead — is the dominant mechanism behind the efficiency gains of the past six years. The evidence is consistent that attention kernels (FlashAttention and its descendants) deliver 2–4x speedups and near-constant memory in sequence length, that fused training-op kernels (Liger and peers) recover roughly 20% throughput and 60% memory on the training side, and that FP8 pipelines now train 10–75% faster with matched quality. Most evidence is preprint-grade and hardware-specific; the field lacks a shared benchmark, and the newest frontier is using LLMs themselves to write the kernels.
Interconnects for large-scale LLM training and inference: RDMA, InfiniBand, NVLink, and the network stack
What does the evidence from 2020 to 2026 show about how interconnect technologies (RDMA, InfiniBand, NVLink, Ethernet/RoCE, and the software stacks built on them) shape the performance, scaling, and cost of large-scale LLM training and inference?
Large-scale LLM training and inference now depend on a hierarchy of interconnects: NVLink and NVSwitch inside servers, InfiniBand or RoCE Ethernet between them, and collective-communication libraries that turn raw fabric into training progress. The 2020-2026 evidence shows the lossless RDMA fabric that carried early GPT-era training is under strain, with congestion-control pathologies and head-of-line blocking documented in both directions, while production operators engineer around it and a lossy, packet-spraying Ultra Ethernet path is emerging. Meanwhile Mixture-of-Experts models and long-context inference have made all-to-all traffic and KV-cache movement the new communication bottlenecks, and NVLink scale-up domains, UALink, and CXL memory pooling are reshaping where computation and memory live. Confidence is moderate: the transport and collective stories are replicated across many independent studies, but the newest hardware (NVLink 4/5, UALink, GB200-class superpods) is covered mainly by vendor-adjacent or preprint evidence.