The LLM inference engine playbook: tricks in vLLM and SGLang

Why this question

The serving engine is where the economics of inference land. A 2x throughput improvement on a serving fleet is worth more than most individual optimizations ever are, and the past four years have produced a dense, fast-moving stack of techniques — the field went from no serving systems to a mature ecosystem in roughly four years ¹⁰¹¹. For anyone doing infrastructure architecture work, this stack is the shared vocabulary: when someone says “chunked prefill”, “radix cache”, “EAGLE”, “PD disaggregation”, or “FP8 KV cache”, they are naming a specific mechanism with a specific bottleneck it attacks, and the ability to map technique to bottleneck is the difference between engineering judgment and cargo-culting.

A companion review on this site already answered “how large are the gains?” across 124 sources (llm-inference-optimization). This review answers a different question: what are the tricks, which papers introduced them, and how are they actually wired into the two engines everyone benchmarks against? It is written to be read top to bottom as a map of the engine, and the final theme is a study path for someone who wants to go from “can describe the tricks” to “can reason about which trick applies where” — including the order that minimizes wasted effort, which is itself a claim the literature supports: gains compose poorly because several techniques attack the same bottleneck ⁵⁰¹⁵.

Scope and methods

The question: which optimization techniques do the vLLM and SGLang serving engines implement, and in what order should an engineer study them? The population is the two engines’ feature catalogs plus the academic sources those features are built on. Included: batching and scheduling, KV-cache management (paging, prefix caching, offloading, quantization), attention kernels and backends, speculative and parallel decoding, inference-time quantization, parallelism (tensor, pipeline, data, expert, context), prefill-decode disaggregation and KV transfer, structured-output decoding, LoRA serving, and MoE/hybrid-architecture serving support. Outcomes are the mechanism of each trick, its originating source, and its status in each engine.

Method, in three waves. First, a full crawl of the current vendor documentation — docs.vllm.ai (v0.27.1 developer preview: features matrix, V1 guide, optimization, attention backends, speculative decoding, quantization, disaggregated prefill, KV offloading, prefix caching, CUDA graphs) and docs.sglang.ai plus the SGLang team blog at lmsys.org/blog — producing a feature inventory of roughly 90 items across both engines. Second, rung-0 reuse of this site’s verified source pool: 1,005 records accumulated across 13 prior reviews, of which 63 were selected here; every reused DOI was re-checked to resolve. Third, verification of 12 candidate arXiv identifiers in-session against their abstract pages (all resolved; 10 became new sources, two were already in the pool). For the 10 new sources, full texts were retrieved from arXiv HTML and load-bearing claims were grep-verified against the text before writing. Screening counts: 12 candidates verified → 10 added; 1,005 pool records → 64 reused; total 74 included, of which 66 full-text and 8 abstract-only.

Deliberate exclusions, per established precedent: Orca (the origin of continuous batching) has no DOI or arXiv version and cannot be a source — its idea is covered through the systems that built on it; GitHub-only artifacts (FlashMLA, DeepEP, DeepGEMM, XGrammar’s repo, LMCache’s repo) are covered through their parent papers and the repositories’ own documentation, with their claims flagged vendor-stated and linked inline; training-side work, datacenter energy, and interconnect hardware are covered by separate reviews on this site. A limitation of method: the vendor documentation is the primary evidence for what each engine currently does, and it is a moving snapshot — the vLLM docs crawled were a developer preview, and feature matrices change between releases.

The landscape

The trick stack has a clear history, and it maps onto the engine codebases. Pre-2023 work established the kernel layer — IO-aware attention ³⁵ and the first serving-friendly quantization schemes ⁵²⁵⁴. 2023 brought the serving systems themselves: vLLM’s PagedAttention and continuous batching ¹, SGLang’s RadixAttention and compressed-FSM decoding ², SARATHI’s chunked prefill ³. 2024 exploded into KV-cache management, speculative decoding families, and disaggregation ⁵⁶⁷⁴⁴. 2025–2026 is consolidation: vLLM rebuilt its scheduler (the “V1” architecture), SGLang added an overlap scheduler, hierarchical caches, and an expanding speculation zoo ⁷⁰⁷¹⁷², and both engines now implement essentially the same catalog. Where they differ is emphasis: vLLM’s V1 unified scheduler treats prefill and decode tokens symmetrically and pushes prefix caching and chunked prefill as defaults; SGLang’s differentiators are the radix-tree cache, compute-communication overlap, and the deepest speculative-decoding integration. The field’s measurement culture is its weakest layer — each paper and each engine benchmarks on its own hardware and workload, and the simulation work that could standardize comparison is only beginning ²¹²².

Theme 1 — The memory wall: why serving is hard

Every trick in this review answers the same underlying fact: autoregressive decode is memory-bound. Generating a token streams the model’s weights and the sequence’s KV state through the GPU, and the bytes fetched per token vastly exceed the arithmetic per token; the roofline-style surveys make this explicit, with attention and linear operators together exceeding 75% of runtime in profiling and a 70B FP16 model needing roughly 140GB of resident state ¹¹. Two consequences follow. First, techniques that cut bytes (quantization, KV compression) and techniques that cut idle time (batching, speculation) dominate; techniques that cut FLOPs alone barely matter at low batch. Second, the two phases of a request have opposite profiles: prefill (prompt processing) is compute-bound and latency-critical, decode (token generation) is memory-bound and throughput-critical — the asymmetry that motivates chunked prefill and disaggregation ⁶⁵. A third, often forgotten layer is queueing: measurements attribute up to 90% of end-to-end latency to waiting in the scheduler, not to the GPU ¹⁹, which is why scheduling discipline is a first-class trick, not a footnote. The surveys that organize this space ¹⁰ and the production-traffic dataset that documents how bursty real workloads are ²² are the two best orientation reads.

Theme 2 — Batching and scheduling: keeping the GPU busy

The first-order fix for an idle GPU is putting more sequences into each forward pass. The decisive innovation was continuous (iteration-level) batching — admitting and retiring requests at every decode step instead of at batch boundaries — introduced by Orca (OSDI’22, unreachable as a citable source; the published record starts with the systems that built on it). vLLM’s PagedAttention paper reported 2–4x throughput over prior systems on top of this scheduler ¹, and SGLang’s paper reports up to 6.4x over prior systems once prefix reuse is added ².

The problem with naive continuous batching is that one long prefill stalls every decode in the batch. Chunked prefill — SARATHI’s “piggybacking” of prefill chunks onto decode slots — reported up to 10x decode-throughput improvement on LLaMA-13B/A6000 ³; Sarathi-Serve generalized it into stall-free batching with multi-step lookahead, reporting 2.6x higher serving capacity on one A100 for Mistral-7B ⁴. Both engines now implement this: in vLLM V1, chunked prefill is enabled by default and the scheduler interleaves prefill and decode at token granularity with a per-request token budget, with max_num_batched_tokens trading time-to-first-token against inter-token latency (vendor docs, optimization); SGLang exposes --chunked-prefill-size and a mixed-chunk mode to co-batch the two phases (vendor docs, server arguments).

Scheduling policy is the next factor. Sequence-length-aware scheduling that groups requests by predicted output length reported 86% throughput gains over vanilla batching ¹²; SGLang’s default policy is longest-prefix-match, which orders work by cache-reuse potential rather than arrival time ². vLLM V1 adds priority scheduling and makes preemption recompute-based (swap-to-CPU was removed in V1, since recomputation is cheaper there) (vendor docs, V1 guide). SGLang’s scheduler runs one batch ahead of the GPU, hiding CPU-side scheduling and radix-cache operations behind the forward pass — its “zero-overhead” batch scheduler, reported at 1.1x throughput in v0.4, and the newer overlap scheduler that also hides KV allocation and metadata preparation (vendor blog, v0.4 announcement and SGLang v0.4+ docs). Multi-step scheduling — running several decode steps per scheduler round — cuts per-step overhead and is how CUDA-graph capture stays efficient ⁴; SGLang exposes it as num-continuous-decode-steps (vendor docs, server arguments).

Two further scheduling tricks sit at the edge: multiplexing multiple models onto one GPU — MuxServe’s spatial-temporal multiplexing reported 1.8x throughput or 2.9x more requests within 99% SLO attainment ¹⁷ — and partitioning a single GPU’s streaming multiprocessors between phases, as DuetServe did with up to 1.3x throughput ¹⁸, an idea SGLang now exposes as PD-multiplexing via NVIDIA GreenContext SM partitioning (vendor blog, PD-Mux).

Theme 3 — The KV cache: paging, reuse, offload

The KV cache grows linearly with context and is why long-context serving runs out of memory long before compute runs out. The foundational trick is PagedAttention: allocate the cache in fixed-size blocks with a block table per sequence, so memory is shared and never fragmented — vLLM’s 2–4x throughput gain over earlier systems came mostly from this ¹. The alternative approach — give each request a contiguous virtual address space and let the driver handle mapping, as vAttention does — reported comparable or better efficiency with lower implementation complexity, at the cost of abandoning the paged kernel ³⁹; both engines still use paging, but the debate is live.

Prefix reuse is the second lever: if a new request shares a prefix with a cached one, the shared KV blocks can be reused instead of recomputed. SGLang’s RadixAttention organizes the cache as a radix tree keyed by token sequences, finds the longest reusable prefix before prefill, and supports LRU/LFU/priority eviction ²; vLLM’s automatic prefix caching hashes KV blocks so any request sharing a prefix hits the cache (vendor docs, APC). The idea generalizes: Prompt Cache reuses attention states of overlapping prompt modules with 8x (GPU) to 60x (CPU) TTFT reductions ²⁶; ChunkAttention makes the attention kernel prefix-aware ²⁵. At fleet scale, the cache becomes a distributed service: LMCache reports up to 15x throughput combined with vLLM ⁸; MemServe’s elastic memory pool reports up to 42% better job completion time ⁹; Infinite-LLM pools KV cluster-wide with 1.35–3.4x gains on contexts up to 2,000K tokens ³⁰; DéjàVu adds fault tolerance by streaming and replicating KV state ²⁹. SGLang’s 2025–2026 additions push this furthest: HiCache extends the radix tree into a three-tier cache (GPU → host → external storage, with file/Mooncake/NIXL/hf3fs backends) (vendor docs, HiCache), and vLLM ships KV offloading to CPU and filesystem tiers with LRU/ARC eviction (vendor docs, KV offloading).

Shrinking the cache is the third lever: quantize it, evict from it, or make it sparse. KIVI’s asymmetric 2-bit quantization reports 2.6x less peak memory with 2.35–3.47x throughput ²⁴; KVQuant reaches under 0.1 perplexity degradation at 3-bit, enabling 1M-token context on a single A100-80GB ²³; both engines now ship FP8 KV caches (per-tensor or per-head scale) as a default-on option (vendor docs, vLLM quantized KV and SGLang quantized KV). Attention sinks — keeping a few initial tokens plus a sliding window — let models stream far beyond their training window with up to 22.2x per-token speedup over recomputation baselines ²⁷. And for the newest sparse-attention models, HiSparse keeps the full KV history in host memory with only a small hot window on the GPU, reporting up to 4.7x peak generation throughput on long-context workloads ⁷⁰; SGLang ships it as a serving mode (vendor docs, HiSparse).

Theme 4 — Attention kernels: making the inner loop fast

Under the scheduler sits the kernel layer. FlashAttention made attention IO-aware — tiling Q/K/V through SRAM so the HBM traffic that dominated attention shrinks by a factor proportional to the tile size — with 2–4x wall-clock speedups and near-linear memory savings ³⁵; FlashAttention-2 rebalanced the work partition across warps and thread blocks ³⁶; FlashAttention-3 added asynchronous pipelining of the matmuls and softmax and low-precision accumulation on Hopper/Blackwell ³⁷. The engines plug these in as selectable backends: vLLM’s attention-backend registry spans FlashAttention 2/3/4, FlashInfer, Triton, FlexAttention, and a family of MLA-specific kernels (vendor docs, attention backends); SGLang exposes the same spread plus per-phase overrides (vendor docs, attention backend).

Three kernel-layer tricks deserve separate mention. FlashInfer showed a customizable attention engine (template-based code generation, batched ragged kernels, fused sampling) cuts inter-token latency 29–69% versus compiler backends ³⁸ — evidence that kernel choice still matters under the systems layer. FlexAttention generalizes attention kernels to arbitrary sparse masks via a single programming model, letting engines generate specialized kernels instead of hand-writing one per pattern ⁴⁰. And multi-head latent attention (MLA) is a model-side trick that became a kernel problem: DeepSeek-V2 compresses the KV cache into a low-rank latent, cutting KV by 93.3% ⁵⁹, which required new kernels — FlashMLA (DeepSeek’s library, GitHub-only; capabilities vendor-stated) and a zoo of MLA attention backends (FlashMLA, CUTLASS_MLA, FlashInfer MLA, TokenSpeed MLA, sparse variants) now listed in both engines’ registries. The decode-phase corollary of FlashAttention — parallelizing over KV blocks across threads and using split-K softmax reduction, the “FlashDecoding” pattern — is folded into the FA2/FlashInfer decode paths and is the reason long-context decode keeps up at all ³⁶³⁸.

Theme 5 — Decoding acceleration: speculative and parallel decoding

Autoregressive generation is serial — one memory-bound step per token. Speculative decoding breaks the serialization: a cheap drafter proposes several tokens, the target model verifies them in one parallel forward pass, and tokens are accepted only where the proposal matches the target’s distribution — which makes the family provably lossless. The two foundational papers established 2–2.5x ⁴² and 2–3x ⁴¹ wall-clock speedups with identical outputs. The drafters evolved fast: tree-based speculation (SpecInfer, 1.5–3.5x) ⁴⁸; Medusa’s multiple decoding heads (2.2x frozen, 2.3–3.6x co-trained) ⁴³; EAGLE’s feature-level autoregression (2.7–3.5x on LLaMA2-Chat-70B) ⁴⁴; EAGLE-2’s dynamic draft trees (3.05–4.26x) ⁴⁵; EAGLE-3’s direct token prediction with multi-layer feature fusion, up to 6.5x ⁴⁶. Draft-free variants exist: lookahead decoding breaks sequential dependency with parallel n-gram guessing (up to 1.8x, 4x multi-GPU) ⁴⁷; REST drafts by retrieving n-grams from a corpus (1.62–2.36x) ⁵¹; TriForce uses the model itself hierarchically for long-sequence generation (2.31x on A100) ⁴⁹.

The 2025–2026 additions shift from draft models to draft heads and draft architectures. Multi-token prediction (MTP) — training the model with several independent output heads — both improves the base model and enables self-speculative decoding at inference ⁶⁷; DeepSeek-V3 embeds MTP heads in the checkpoint and reports about 1.8x decoding speedup from them ⁶⁰. Both engines serve MTP natively — SGLang reports up to 60% higher output throughput on DeepSeek-V3 via its MTP path (vendor docs, speculative decoding and MTP blog). DFlash drafts a whole block with a lightweight diffusion model in one forward pass ⁷²; DSpark combines a parallel draft backbone with a lightweight sequential module (semi-autoregressive drafting) and schedules verification length per request by estimated acceptance probability ⁷¹; FR-Spec compresses the draft search to a frequency-ranked vocabulary subset, cutting LM-head computation 75% on 128K-vocabulary models ⁷³. Both engines now support the full ladder — draft models, EAGLE/EAGLE-2/EAGLE-3, MTP, n-gram prompt lookup, suffix matching, PARD, plus dynamic speculation length that tunes the draft count per batch size (vendor docs, vLLM speculative decoding and SGLang’s speculative decoding).

The load-bearing caveat is the interaction with batching: speculation’s benefit shrinks as batch size grows, and the explicit synergy study shows optimal speculation length decreases with batch size, with adaptive strategies adding about 9% latency reduction on time-varying loads ⁵⁰. Speculative decoding is a low-batch, low-latency technique — at production batch sizes the target model’s own batched compute dominates. This is the single most important fact to internalize before deploying any drafter.

Theme 6 — Quantization: fewer bytes per weight

Quantization attacks the largest byte flow in decode — the weights — and is the most mature family. The engine-relevant ladder: GPTQ introduced layer-wise second-order quantization (3.25x A100 / 4.5x A6000 end-to-end speedups on 175B models) ⁵²; AWQ protects the ~1% salient activation channels via per-channel scaling (3–4x speedup; 4-bit 70B on mobile hardware) ⁵³; SmoothQuant moved the activation outliers into the weights so W8A8 becomes viable (1.56x speedup, 2x memory reduction, serving a 530B model) ⁵⁴; the FP8 formats paper established the two 8-bit float encodings (E4M3/E5M2) that now dominate frontier serving ⁵⁵; and system co-design matters as much as the arithmetic — QServe’s W4A8KV4 reports 2.4x (A100) and 3.5x (L40S) throughput over TensorRT-LLM ⁵⁶, and Atom’s fine-grained mixed-precision 4-bit reports up to 7.7x over FP16 at equal latency ⁵⁷. DeepSeek-V3 made FP8 the production format for a 671B model, training and serving with FP8 weights and activations ⁶⁰.

The engines integrate this as a support matrix rather than inventing new schemes: both serve GPTQ, AWQ, FP8 (W8A8), INT4/INT8, GGUF, bitsandbytes, and the newer MXFP4 and NVFP4 formats, with Marlin-family kernels for the 4-bit GEMMs; FP8 KV cache and FP8 attention are engine features, and quantization plugins are registerable (vendor docs, vLLM quantization and SGLang quantization). The practical ranking for a server engineer: FP8 is nearly free, 4-bit weight-only is the current frontier for capacity, and sub-4-bit remains accuracy-dependent — the independent quantization benchmark shows the trade-offs are real and context-dependent ⁷⁴¹¹, and KV quantization (Theme 3) is a separate axis that multiplies with weight quantization rather than replacing it.

Theme 7 — Parallelism: TP, PP, DP, EP, CP

When one GPU is not enough, the model splits across many. Tensor parallelism shards every weight matrix across GPUs (the default for single-node serving); pipeline parallelism splits layers, with SGLang’s chunked pipeline parallelism reporting 3.31x prefill throughput (PP4xTP8 vs TP8) and up to 67.9% TTFT reduction on DeepSeek-V3.1/H20 (vendor blog, chunked pipeline). For MoE models, expert parallelism is the load-bearing trick: experts are sharded across GPUs and tokens are routed, turning the all-to-all into the main cost — DeepSpeed-MoE reported 7.3x better latency/cost than prior MoE inference ⁵⁸, and the DeepSeek recipe (EP + communication overlap via DeepEP/NIXL, vendor-stated) is how 671B models are served at all ⁵⁹⁶⁰. MegaScale-Infer pushes further with disaggregated expert parallelism and shared KV stores ³². Data parallelism replicates the model and shards the batch; SGLang’s data-parallel attention (DPA) is the refined version for MLA models — replicating the attention shards instead of TP-splitting MLA’s single KV head, which avoids KV duplication and was reported at up to 1.9x decode throughput on DeepSeek-class models (vendor docs, DPA guide). SGLang’s production router (SMG) adds cache-aware request routing across replicas, reported at +92% throughput via 20%→75% cache-hit rates (vendor docs, SMG). Context parallelism splits the sequence dimension for long contexts — LoongServe’s elastic sequence parallelism adjusts parallelism per request and phase, reporting 3.85x over chunked prefill and 5.81x over prefill-decode disaggregation ³¹. And for the hybrid architectures arriving in 2026 (linear-attention states alongside KV), both engines are adding dedicated kernels and state management — the Mamba line established the linear-attention efficiency argument ⁶², and SGLang’s GDN/Mamba support manages those states in the same page-based memory system (vendor docs, attention backend).

Theme 8 — Disaggregation: separating prefill from decode

Prefill is compute-bound and latency-critical; decode is memory-bound and throughput-critical. Colocated, they interfere: measurements show naive mixing can cause 10x prefill slowdowns, 5x prefill-decode slowdowns, and 16% decode throughput loss ²⁰. Disaggregation runs them on separate machines, each tuned for its phase. Splitwise reported 1.4x throughput at 20% lower cost (2.35x under equal power and cost budgets) ⁵; DistServe’s phase-specific parallelism reported serving 7.4x more requests or 12.6x tighter SLOs ⁶. The hidden cost is moving the KV cache between phases: KVDirect showed per-request latency can be cut 55% by pull-based, tensor-centric transfer ³³; SpectrumKV showed transfer precision matters — mixed FP16/INT8/INT4 transfer keeps perplexity change at +1.97% versus +25.85% for a naive baseline at a 50% transfer budget, with 50–62% TTFT reduction ³⁴; CacheGen compresses KV states for transmission with 3.5–4.3x smaller caches ²⁸. The KVCache-centric architecture — Mooncake’s tiered CPU/DRAM/SSD store reported up to 525% simulated throughput gains under SLOs and 75% more requests handled by Kimi on real workloads ⁷ — is now the reference design, and both engines implement it: vLLM’s disaggregation is connector-based (LMCache, NIXL, Mooncake, FlexKV, offloading connectors; the docs are explicit that disaggregation improves tail latency and phase tuning, not raw throughput) (vendor docs, disagg prefill); SGLang ships prefill/decode instance modes with Mooncake or NIXL transfer, decode-side radix caching, and even a full three-way EPD split (encoder-prefill-decode) for vision-language models, whose paper reports up to 71% TTFT reduction and 90–100% SLO improvement ⁶⁸⁷⁰ (vendor docs, PD disaggregation and EPD).

The scheduling layer of disaggregation is where the newest gains sit: Preble’s distributed prompt scheduling targets p99 latency for short-output workloads ⁶⁶; LAPS’s length-aware prefill reduces prefill latency over 30% versus vanilla SGLang under disaggregation ¹³; dynamic routing of prefill-capable decoders cuts turn-2+ TTFT by ~68% ¹⁴. And the literature is now questioning the binary split itself: TaiChi unifies aggregation and disaggregation with latency shifting, improving goodput up to 77% under balanced SLOs ¹⁶, and simulation work shows the split’s value depends on workload mix and hardware generation — disaggregation raises serving throughput by up to 75% on current GPUs, but “when does disaggregation pay?” is the right question, not “always disaggregate” ¹⁵.

Theme 9 — Serving-side features: structured outputs, LoRA, multimodal

A serving engine’s tricks are not only about speed — some features make the product possible at all. Constrained decoding (JSON, regex, grammars) is the clearest case: the FSM formulation of guided generation ⁶⁹ and SGLang’s compressed-FSM trick (state deduplication for ~3x faster JSON decoding) ² made structured outputs cheap; XGrammar’s byte-level pushdown automaton then made context-free grammars fast — up to 100x per-token latency reduction versus prior engines, near-zero overhead in production — and is now the default structured-output backend in both vLLM and SGLang ⁶⁴ (vendor docs, vLLM structured outputs and SGLang structured outputs). SGLang additionally ships jump-forward decoding for deterministic grammar transitions (vendor docs, same page) and grammar suspension inside reasoning-model thought spans.

LoRA serving is the multi-tenant trick: many small adapters over one base model. S-LoRA’s unified paging and heterogeneous batching kernels serve thousands of adapters ⁶³; Punica’s segmented gather matrix-vector kernel batches many adapters into one CUDA kernel, reporting 12x throughput over prior serving systems at +2ms/token ⁶⁵. Both engines support request-level adapter swapping with fused batched LoRA compute (vendor docs, vLLM LoRA and SGLang LoRA). Multimodal serving adds encoder work to the pipeline: batch-level data parallelism for encoders (~10% TTFT gain at TP=8 in vLLM), processor and IPC caching so images are not re-encoded across turns (vendor docs, vLLM optimization), and the EPD disaggregation pattern for separating encoder GPUs entirely ⁶⁸. Minor but real: both engines now offer faster tokenizer backends (vLLM’s fastokens, vendor-stated) and streaming detokenization, and vLLM V1 removed per-request logit processors in favor of global ones — a reminder that scheduling simplification is itself a performance trick (vendor docs, V1 guide).

Theme 10 — MoE and inference-cost architectures

The engine’s tricks are bounded by the model’s architecture, and the frontier architectures of 2024–2026 were built for serving. Mixtral showed a sparse MoE can match a dense model 5x its active size ⁶¹; DeepSeek-V2 combined fine-grained expert specialization with MLA, cutting the KV cache 93.3% and reporting 5.76x maximum generation throughput over its dense predecessor — over 50K generation tokens/s on one 8x-H800 node ⁵⁹; DeepSeek-V3 scaled that to 671B total / 37B active with FP8 arithmetic, MTP heads, and auxiliary-loss-free routing ⁶⁰. Serving these well is an engine problem: expert parallelism plus communication-computation overlap ⁵⁸³², plus a run of MoE-specific tricks now in the engines — fused shared-expert dispatch (“waterfill”), expert load balancing, redundant experts, and CPU offloading of experts with AMX kernels (KTransformers) (vendor docs, SGLang expert parallelism and server arguments). Hybrid linear-attention models (Mamba-style) trade exact attention for constant-memory state ⁶²; the engines’ treatment of hybrid states inside the paged memory system is the 2026 frontier of this theme.

Theme 11 — A self-study path through the engine, in order

The review above is the map; this is the route. The ordering below follows the dependency structure of the field — each stage’s material assumes the previous stage’s — and each stage ends with a checkpoint: a question you should be able to answer precisely, and a hands-on experiment that proves it. Expect to spend 1–2 focused weeks per stage if you are doing the experiments; the reading alone is faster.

Stage 0 — The roofline arithmetic (1–2 days). Read the roofline survey’s core sections ¹¹ and the serving survey’s framing ¹⁰. Then do the arithmetic yourself: for a 7B FP16 model, compute bytes-per-token for weights (14GB streamed per token at batch 1) and KV (2 × layers × heads × head_dim × 2 bytes × context), and compare with a GPU’s HBM bandwidth to derive the memory-bound tokens/s ceiling. Checkpoint: explain why decode is memory-bound and prefill is compute-bound, with numbers.

Stage 1 — The serving problem and the first systems (3–5 days). Read the PagedAttention/vLLM paper ¹ and the SGLang paper ² — these two are the center of gravity everything else cites. Hands-on: run vLLM on a small model, then SGLang; measure TTFT, inter-token latency, and throughput at batch 1 vs a concurrent load of 16; watch what continuous batching does to the scheduler logs. Checkpoint: explain paging, iteration-level batching, and prefix caching to someone else without notes.

Stage 2 — Scheduling and memory management (3–5 days). Read SARATHI ³, Sarathi-Serve ⁴, and the sequence-scheduling paper ¹². Hands-on: toggle chunked prefill and prefix caching in vLLM (they are defaults in V1 — turn them off to feel the difference); serve a shared-prefix workload and watch TTFT collapse with prefix caching; try the LPM scheduler in SGLang. Checkpoint: explain why chunked prefill exists, what it trades, and when prefix caching pays.

Stage 3 — Kernels (3–5 days). Read FlashAttention 1, 2, 3 in order ³⁵³⁶³⁷ — the IO-awareness argument is the intellectual core — then FlashInfer for the engine-integration view ³⁸. Hands-on: profile a small model’s decode with ncu/nsight, identify the memory-bound GEMMs, and confirm attention shows up as bandwidth-bound; try the different attention backends the engines expose and compare. Checkpoint: explain tiling, the online softmax trick, and why kernel choice changes serving latency.

Stage 4 — Speculative decoding (3–5 days). Read the two foundation papers ⁴¹⁴², then EAGLE ⁴⁴, EAGLE-2 ⁴⁵, EAGLE-3 ⁴⁶, and — before touching a deployment — the batching-synergy paper ⁵⁰. Hands-on: enable EAGLE or MTP in SGLang at batch 1 and at batch 64; the curve you measure is the whole lesson. Checkpoint: explain why speculation stops helping at high batch, and why the lossless guarantee is about distribution, not quality.

Stage 5 — Quantization (3–5 days). Read SmoothQuant ⁵⁴, GPTQ ⁵², AWQ ⁵³, and the FP8 formats paper ⁵⁵. Hands-on: quantize a model with AWQ or FP8, serve it, and compare output quality and tokens/s against FP16; enable the FP8 KV cache and watch the context-length ceiling move. Checkpoint: explain the difference between weight-only and W8A8, and when each is the right choice.

Stage 6 — Distribution and disaggregation (4–7 days). Read Splitwise ⁵, DistServe ⁶, Mooncake ⁷, and the “when does disaggregation pay” simulation ¹⁵ — read that one before you build anything. Hands-on: run tensor parallelism (TP=2 vs TP=1), then stand up the engines’ PD-disaggregation demo and watch TTFT/ITL decouple. Checkpoint: explain the prefill/decode resource asymmetry, what KV transfer costs, and the conditions under which disaggregation is a net win.

Stage 7 — Frontier architectures (3–5 days). Read DeepSeek-V2 ⁵⁹ and DeepSeek-V3 ⁶⁰ for the MLA/MTP/FP8/EP combination, HiSparse for the sparse-attention serving pattern ⁷⁰, and one of the new drafter papers (DSpark ⁷¹ or DFlash ⁷²) to see where speculation is going. Hands-on: serve a DeepSeek-class model with the engines’ recommended recipe (TP + EP + DPA + MTP) and measure. Checkpoint: explain MLA’s KV reduction, MTP’s speedup mechanism, and why MoE serving is an all-to-all problem.

Stage 8 — Synthesis and measurement discipline (ongoing). Read the simulation ²¹ and workload-dataset ²² work — the field’s answer to its comparability problem — and the composition caveat ⁵⁰¹⁵. Capstone: write a one-page design for serving a specific model on specific hardware, stating which tricks you would enable, which you would not, and the bottleneck each targets; then measure the real system and reconcile. If tokens-per-watt is a goal, the companion review on this site (energy efficiency) covers that axis. Checkpoint: defend every trick in your design by name of the bottleneck it attacks.

Where the evidence disagrees

The size of speculation’s gain. Reported speedups span 1.5x to 6.5x for the same underlying idea ⁴²⁴⁶. The resolution is batch size and drafter quality: at batch 1 with a well-matched drafter, 3–6.5x is reproducible; at production batch sizes the gain collapses because the batched forward pass dominates ⁵⁰. “Speculative decoding gives 2–3x” and “speculative decoding barely helps in production” are both true, at different operating points — and the engines’ spec-decode docs (which recommend dynamic/draft-count tuning) implicitly concede this.

Disaggregation’s universality. Splitwise, DistServe, and Mooncake show large gains ⁵⁶⁷; the simulation and KV-transfer literature shows the gains depend on workload mix, hardware generation, and the cost of moving KV state ¹⁵³³³⁴. vLLM’s own docs now state disaggregation does not improve raw throughput — the field is moving from “disaggregate everything” to “disaggregate when the phase ratio and transfer cost justify it”, and the later position is better supported.

Paged vs virtual-memory attention. vLLM’s paging eliminated fragmentation ¹; vAttention argues the paged kernel is the wrong abstraction and contiguous virtual memory with driver-level mapping is simpler and faster ³⁹. Both engines page today, but the argument is not settled, and it will resurface as sparse-attention models change what the cache holds ⁷⁰.

Vendor numbers vs paper numbers. The engines report headline gains (“15x with LMCache”, “525% simulated” ⁸⁷) that are not reproducible from the papers alone, because there is no shared benchmark ²¹. This is not fraud — it is the field’s measurement culture — but it means vendor-stated figures should be read as directional, and the papers’ “up to” figures as ceilings.

Gaps and open questions

Composition. Production systems stack batching, prefix caching, speculation, quantization, and disaggregation simultaneously, yet nearly every paper optimizes one layer in isolation. The interaction effects are known to be non-additive in at least one direction — speculation shrinks with batching ⁵⁰ — and the engines’ defaults encode untested assumptions about the joint space. What would settle it: a published per-trick ablation on identical hardware.

No shared benchmark. The single most important gap for a practitioner choosing between tricks: no canonical hardware, workload, and metric set exists ²¹²². The vendor docs make this worse by reporting throughput on their own models and GPUs.

Doc-vs-paper drift. The engines move faster than the literature: features like DPA, HiCache, SMG routing, and adaptive verification are documented only in vendor blogs and code, with no citable paper (or papers that trail the implementation by months). A review like this one is therefore a snapshot; the vendor links are the only stable record.

Long-context integration. Million-token serving exists on separate axes — quantized cache ²³, sparse attention ⁷⁰, cluster parallelism ³¹ — but a system combining all three under production scheduling has not been demonstrated in the retrieved literature.

Confidence and limitations

Confidence is moderate, with two tiers. The mechanism layer — what each trick does and which bottleneck it attacks — is well established and replicated across many independent groups; the paper claims behind the catalog were verified in this review or in the companion deep review (llm-inference-optimization), whose 124 sources were extracted and spot-checked against full texts. The implementation layer — exactly what each engine does today — rests on vendor documentation (vLLM docs at v0.27.1 developer-preview, SGLang docs and blogs as crawled 2026-08-18) and is vendor-stated; feature matrices change between releases and the preview may not match the stable release.

This review’s own limitations: 8 of 74 sources were abstract-only (mostly ACM/ACL records whose full texts were not openly retrievable in-session); Orca, the origin of continuous batching, is absent for lack of a DOI; GitHub-only artifacts (FlashMLA, DeepEP, NIXL, LMCache’s implementation) are covered through parent papers and repository documentation with their numbers flagged vendor-stated; the corpus is English and arXiv-heavy; and the engine-crawl cutoff (2026-08-18) will age quickly in a field that ships weekly.