On this page

Summary
Why this question
Scope and methods
The landscape
Theme 1 — Serving systems: batching, chunking, and scheduling
Theme 2 — The KV cache: paging, eviction, compression, reuse
Theme 3 — Quantization: cutting bytes per weight
Theme 4 — Speculative and parallel decoding
Theme 5 — Prefill-decode disaggregation and memory pooling
Theme 6 — Parallelism for long context
Theme 7 — Mixture-of-experts and inference-cost architectures
Theme 8 — Memory-limited deployment: offloading and consumer hardware
Where the evidence disagrees
Gaps and open questions
Confidence and limitations
Evidence table
References

Optimizing large language model inference

Which techniques most improve the throughput, latency, and cost of large language model inference?

Updated: 8 Aug 2026
Sources: 124
Years: 2018–2026
Confidence

https://reviews.lewiswon.me/reviews/llm-inference-optimization/ · Updated 8 Aug 2026

LLM inference serving systems KV cache quantization speculative decoding mixture-of-experts prefill-decode disaggregation

How this review was made

Databases: OpenAlex, Crossref, arXiv, Semantic Scholar, DBLP
Queries (literal): continuous batching LLM serving; chunked prefill LLM inference; KV cache management LLM serving; KV cache eviction long context; KV cache quantization compression; prefix caching LLM serving; speculative decoding large language model; parallel decoding LLM generation; weight quantization large language model inference; activation quantization LLM inference; mixture of experts serving efficiency; expert parallelism inference; prefill decode disaggregation LLM serving; sequence parallelism long context; ring attention long sequence transformer; offloading large language model single GPU inference; grouped query attention inference; multi head latent attention; attention sink streaming inference; high throughput LLM inference serving; efficient inference large language model survey; draft model speculative inference verification
Search last run: 2026-08-08
Screening: 124 sources used · 2018–2026 · deep review

Summary

The short version

Optimizing large language model (LLM) inference is, at bottom, a memory problem: autoregressive generation streams weights and key-value state through a GPU that can compute much faster than it can fetch bytes, so techniques that cut bytes (quantization, KV-cache compression) or cut idle time (batching, scheduling, speculative decoding) dominate the reported gains. Across 124 sources, the consistent pattern is that serving systems deliver 2–6x throughput improvements, KV-cache techniques 2–5x in memory and often 2–3x in speed, quantization 1.5–4x in throughput with modest quality cost at 4-bit, and speculative decoding 2–3.5x in latency with the strongest methods reaching 4–6.5x. The field’s confidence is moderate: the mechanics are well understood and replicated across many independent groups, but almost every headline is an “up to” figure on the authors’ own hardware and workload, and there is no shared benchmark that would let a reader compare techniques across papers. The single most important caveat is that gains compose poorly — several optimizations target the same bottleneck, so combining them yields less than the product of their individual speedups.

Why this question

Inference, not training, is where the cost of serving LLMs lands: every token generated for every user runs through the same model, and unlike training, inference is latency-sensitive and runs continuously. The economics are large enough that a 2x throughput improvement on a serving fleet is worth more than most academic optimizations ever are, which is why this literature is one of the most active in systems research — the field went from no serving systems to a mature ecosystem of them in roughly four years, with survey after survey documenting the expansion ⁶⁹⁶⁷⁸⁰.

The question is also the technical core of infrastructure-architecture work, where the ability to reason about which bottleneck a technique attacks — memory bandwidth, GPU idle time, serial dependency, or memory capacity — is the difference between engineering judgment and cargo-culting. Roofline-style analyses make the framing explicit: LLM inference is memory-bound, with attention and linear operators together exceeding 75% of runtime in profiling, and a 70B model in FP16 needing roughly 140GB of memory ⁷⁹⁶⁶. This review maps the technique families to the bottlenecks they address and reports what the evidence actually shows about the size of each gain.

Scope and methods

The research question: which techniques most improve the throughput, latency, and cost of LLM inference, and how large are the gains? The population is systems and algorithms for serving autoregressive decoder-only transformer LLMs. Included: serving systems and scheduling, KV-cache management (paging, eviction, compression, reuse), inference-time quantization, speculative and parallel decoding, prefill-decode disaggregation, sequence/context parallelism, MoE inference optimization, offloading, and architecture changes whose explicit purpose is inference cost (multi-query/grouped-query attention, latent attention, attention sinks, selective state spaces). Outcomes are throughput (tokens/s), latency (time-to-first-token, time-per-output-token, job completion time), memory footprint, and cost.

Searches ran 2026-08-08 across OpenAlex, Crossref, arXiv, Semantic Scholar, and DBLP: 22 literal theme queries plus 80 landmark-title lookups and citation-graph snowballing from eight anchor papers, merged to a pool of 1,380 deduplicated records (OpenAlex daily budgets were exhausted mid-run, so exact-title resolution fell to arXiv and Crossref ladders; every remembered arXiv identifier was verified against its abstract page — seven of seven remembered IDs in one pass pointed at unrelated papers). Screening: 1,380 retrieved → deduplicated to 1,380 → 124 included. Excluded: kernel-level CUDA/Triton fusion (covered by a separate review on this site), datacenter energy (separate review), interconnect hardware (separate review), training-focused work, USENIX-only papers without DOIs (Orca, the origin of continuous batching, is OSDI’22 with no DOI or arXiv version and had to be dropped — its results are discussed here only through the systems that built on it), and one paper whose abstract was unreachable (TwinPilots, SoCC’24). Every DOI was verified to resolve; 96 of 124 sources were read in full text, the remainder abstract-only. Evidence was extracted per source before synthesis, and load-bearing numbers were spot-checked against full texts.

The landscape

This is a young, explosive field: of the 124 included sources, 61 are from 2024 alone, 35 from 2023, and the corpus spans 2018–2026. The work comes overwhelmingly from ML-systems research groups publishing on arXiv first and at systems venues later; 91 of 124 sources are preprints or preprint-first, and only 33 carry a peer-reviewed venue record. Two consequences follow. First, the field’s “consensus” is largely a preprint consensus — high-quality, but rarely independently replicated before claims propagate. Second, the measurement culture is the weakest part of the literature: each paper benchmarks on its own GPUs, models, and workloads, with no shared harness. Vidur, a simulator, reports it can estimate latency within 9% error and replace a 42K-GPU-hour deployment search with a one-hour CPU run ⁷⁷ — precisely the reproducibility gap the field needs to close. BurstGPT, a public 10.31M-trace production workload dataset spanning 213 days, is the main public counterweight to private traces, and it documents the bursty, uneven request patterns that make serving optimization hard in practice ⁷⁸.

The shape of the literature follows the bottleneck: pre-2023 work (multi-query attention, blockwise decoding, early quantization) attacked the memory wall directly; 2023 brought the serving systems; 2024 exploded into KV-cache management, speculative decoding families, and disaggregation; 2025–2026 is consolidating on long-context serving, MoE systems, and the interaction of techniques. Compression-oriented surveys organize the space into model-, data-, and framework-centric categories ⁶⁷⁶⁸, while serving-focused surveys catalog post-2023 system optimizations that leave the decoding mechanism untouched ⁸⁰.

Theme 1 — Serving systems: batching, chunking, and scheduling

The first-order fix for idle GPUs is batching more sequences into each forward pass, and the decisive innovation was continuous (iteration-level) batching, introduced in the Orca serving system at OSDI’22. Because Orca itself is unreachable as a citable source (no DOI), the published record of the idea starts with the systems that built on it. PagedAttention/vLLM reported 2–4x throughput over prior systems by eliminating KV-cache fragmentation with paged, non-contiguous cache allocation, and — critically for the field’s trajectory — released as open source, making it the common baseline every later paper measures against ¹. Sarathi’s chunked prefill — interleaving prefill chunks with decodes so the GPU never idles on one long prompt — reported up to 10x decode-throughput improvement on LLaMA-13B/A6000 and 6.29x pipeline-bubble reduction ²; Sarathi-Serve extended this to stall-free batching with 2.6x higher serving capacity on one A100 for Mistral-7B and up to 5.6x end-to-end with pipeline parallelism ³.

Scheduling is where the next factor lives. FastServe’s skip-join multi-level feedback queue reports up to 31.4x and 17.9x throughput over vLLM under matched average and tail latency, exploiting that up to 90% of end-to-end latency is queueing ¹³; its numbers are the largest single-system gain in the serving literature, and they are throughput-at-fixed-latency, not latency gains. Sequence-length-aware scheduling that groups requests with similar predicted output lengths reports 86% throughput gains over vanilla batching ⁹⁶; speculative shortest-job-first with a light proxy model for output-length prediction reports 30.5–39.6% lower average job completion time and 2.2–3.6x throughput over FCFS schedulers ⁹⁵. MuxServe’s spatial-temporal multiplexing — partitioning one GPU across models via CUDA-MPS and colocating popular models — reports 1.8x throughput or 2.9x more requests within 99% SLO attainment ⁷. The SGLang system contributed RadixAttention, prefix-aware KV reuse plus compressed finite-state-machine decoding for structured outputs, reporting up to 6.4x higher throughput than prior systems across LLM and multimodal workloads ⁸. Multi-tenant LoRA serving (S-LoRA) reports up to 4x throughput over naive adapter serving via unified paging and heterogeneous batching kernels ¹⁴. Below the systems, FlashInfer reports 29–69% inter-token latency reduction from a customizable attention engine versus compiler backends, with 13–17% speedups for parallel generation — evidence that kernel-level gains still matter under the systems layer ⁸⁹. DeepSpeed-Inference’s analytical partitioning plus low-level optimizations reached 29ms per token at low batch and 76% MFU at large batch on PaLM 540B, establishing that with enough batching, compute efficiency becomes reachable ¹². Fine-grained GPU-CPU workload allocation for decoder models adds up to 105% throughput on commodity hardware ¹¹⁶.

Theme 2 — The KV cache: paging, eviction, compression, reuse

The KV cache grows linearly with context length and is the reason long-context serving runs out of memory long before compute runs out. vLLM’s PagedAttention attacked fragmentation at the page level ¹. The next layer of work attacks the cache’s content. Eviction methods argue most cached tokens barely matter: H2O keeps a 20% “heavy-hitter” budget and reported up to 29x throughput over DeepSpeed Zero-Inference and Hugging Face Accelerate baselines ¹⁵; Scissorhands, built on the persistence-of-importance hypothesis, reported 5x memory reduction (up to 20x combined with 4-bit quantization) ¹⁶; SnapKV reported 3.6x faster generation and 8.2x memory efficiency, decoding under 40ms/token at 16K context and handling 380K-token contexts on a single A100-80GB ¹⁷; PyramidKV, which allocates cache budget by layer, matches full-cache performance at 12% of the cache and reports 100.0 accuracy on needle-in-a-haystack with only 128 entries ⁷¹; FastGen compresses 35% of the cache while recovering over 95% of attention scores via per-head eviction patterns ¹⁰⁰; NACL reports 80% and 76% improvements on short- and long-text tasks over eviction baselines at 50% cache ¹⁰¹; PyramidInfer reports 2.2x throughput with over 54% KV GPU-memory reduction ¹⁰⁶. A related line shows the cache is redundant across heads: DuoAttention applies full caching only to retrieval heads and a constant-length cache to streaming heads, cutting long-context memory up to 2.55x ⁹⁷; Layer-Condensed KV caching computes keys and values for only a few layers and reports up to 26x higher throughput ¹⁰⁷. SparQ Attention instead fetches only the most relevant cached history, cutting attention data transfers up to 8x with no fine-tuning ⁹⁹.

Quantizing the cache is orthogonal to evicting it: KIVI’s asymmetric 2-bit quantization reports 2.6x less peak memory (including weights) with 2.35–3.47x throughput ¹⁹; KVQuant reaches under 0.1 perplexity degradation at 3-bit, enabling 1M-token context for LLaMA-7B on a single A100-80GB ²⁰; GEAR combines quantization with low-rank plus sparse error correction for near-lossless 4-bit compression at up to 2.38x throughput and 2.29x peak-memory reduction ²¹. CacheGen compresses KV states for transmission, reducing cache size 3.5–4.3x and fetch delay 3.2–3.7x ²⁵. A comprehensive benchmark of ten-plus methods, with an aligned evaluation environment across seven task categories, concluded that the trade-offs are real and context-dependent — a rare independent check on a literature of self-reported numbers ¹⁰⁹.

Reuse is the third lever: Prompt Cache reuses attention states of overlapping prompt modules with 8x (GPU) to 60x (CPU) TTFT reductions ²²; CacheBlend reuses non-prefix chunks selectively, reducing TTFT by 2.2–3.3x and increasing throughput 2.8–5x versus full recompute ²³; RAGCache organizes retrieved-knowledge states in a knowledge tree across GPU and host memory with up to 4x TTFT reduction for retrieval-augmented workloads ²⁴; ChunkAttention makes attention prefix-aware with 3.2–4.8x kernel speedups ¹⁰⁸. DroidSpeak shows KV caches can even be shared across different models of the same architecture — up to 4x throughput by reusing another model’s cache ⁹². InfiniGen speculatively prefetches only the tokens the model will actually attend to, improving offloading-based long-text inference up to 3.00x ⁹⁰. StreamingLLM established that models trained on finite windows can stream far beyond them by keeping a few attention-sink tokens plus a sliding window, with up to 22.2x per-token speedup over recomputation baselines ¹⁸, and later work showed attention sinks are a learned phenomenon — they emerge only after effective optimization on sufficient data, and disappear when softmax normalization is replaced with sigmoid attention ⁹⁸.

Theme 3 — Quantization: cutting bytes per weight

Quantization attacks the largest byte flow in decoding — model weights — and is the most mature family. The evidence is consistent: 8-bit is essentially free (LLM.int8() halves inference memory with full-precision performance, computing over 99.9% of values in 8-bit and enabling 175B models on a single server ²⁹; SmoothQuant’s W8A8 reports up to 1.56x speedup and 2x memory reduction with negligible accuracy loss, serving a 530B model ²⁸), and 4-bit is the current practical frontier. GPTQ quantizes 175B models in about four GPU hours with 3.25x (A100) to 4.5x (A6000) end-to-end speedups ²⁶; AWQ’s activation-aware scaling of the 1% salient weight channels reports 3–4x speedup and puts a 4-bit 70B model on mobile hardware ²⁷; SqueezeLLM’s sensitivity-based non-uniform 3-bit quantization halves the perplexity gap to FP16 versus state-of-the-art methods at equal memory ³⁸; SpQR isolates outlier weights and runs a 33B model on a 24GB GPU with 4x memory compression ³⁹; OmniQuant quantizes the LLaMA-2 7-70B family on a single A100 in 1–16 hours with 128 samples ³⁷. Scaling-law evidence across 35,000+ experiments concludes 4-bit precision is almost universally optimal for total model bits versus zero-shot accuracy ⁴⁰. ZeroQuant’s end-to-end INT8 achieves up to 5.19x speedup on BERT/GPT-3-style models ³⁰.

Below 4 bits the story is harder: QuIP’s incoherence processing plus adaptive rounding made 2-bit quantization viable for the first time ³¹; QuIP# (Hadamard incoherence processing plus E8-lattice codebooks and fine-tuning) reached state of the art at 4 bits and below ³²; SpinQuant’s learned rotation matrices narrow the W4A4KV4 accuracy gap to 2.9 points on LLaMA-2 7B ³³; BitNet explores 1-bit weights with competitive quality and a scaling law ⁴¹; Outlier Suppression+ reports near-floating-point performance at 8-bit and 6-bit standard quantization and a 15.5% state-of-the-art gain at 4-bit ⁷⁵. System co-design matters as much as the arithmetic: QServe’s W4A8KV4 quantization reports 2.4x (A100) and 3.5x (L40S) throughput over TensorRT-LLM on Qwen1.5-72B, cutting serving cost 3x ³⁵, and Atom’s mixed-precision fine-grained 4-bit reports up to 7.7x throughput over FP16 at the same latency target ³⁶. FP8 formats for weights and activations match 16-bit training results on models up to 175B parameters and preserve accuracy where INT8 post-training quantization fails ³⁴. The LLMC benchmark provides an independent, calibration-aware comparison across methods and formats ¹¹⁰, and the CPU-specific result — automatic INT4 weight-only quantization with 20–80ms per-token latency for 6B–20B models within 1% of FP32 ¹⁰³ — shows quantization’s reach beyond GPUs.

Theme 4 — Speculative and parallel decoding

Autoregressive generation is serial: one token at a time, each step memory-bound. Speculative decoding breaks the serialization by having a cheap drafter propose several tokens and the target model verify them in parallel, accepting the proposal only where it matches the target’s distribution — which makes the family provably lossless. The two foundational papers established 2–2.5x ⁴² and 2–3x ⁴³ wall-clock speedups with identical outputs. The drafters evolved quickly: SpecInfer’s tree-based speculation reports 1.5–2.8x for distributed and 2.6–3.5x for offloading-based inference ⁴⁹; Medusa’s multiple decoding heads 2.2x with a frozen backbone and 2.3–3.6x with co-training ⁴⁴; EAGLE’s feature-level autoregression 2.7–3.5x on LLaMA2-Chat-70B ⁴⁵; EAGLE-2’s dynamic draft trees 3.05–4.26x ⁴⁶; EAGLE-3, using direct token prediction with multi-layer feature fusion, up to 6.5x — about 1.4x over EAGLE-2, and 1.38x throughput in SGLang at batch 64 ⁷⁶. Recurrent Drafter reports 2.8x on H100 using an RNN drafter conditioned on the target’s hidden states ⁷⁴. Draft-free variants exist: blockwise parallel decoding (up to 2x fewer iterations, 4x wall-clock with relaxed verification) ⁴⁷; lookahead decoding (up to 1.8x, 4x with multi-GPU scaling) ⁷²; REST, retrieval-based (1.62–2.36x) ⁴⁸; TriForce, hierarchical self-speculation (2.31x on A100, 7.78x in offloading) ⁵²; and self-speculative approaches that draft by skipping layers or stages: LayerSkip up to 2.16x ¹¹², Draft & Verify up to 1.99x ¹²³, staged speculation 3.16x on a 762M model ⁵⁰, and cascade drafting adding up to 81% speedup over base speculation ⁵¹. Big Little Decoder achieves 2.12x with a small quality cost ¹²⁴, ProPD’s dynamic token-tree generation outperforms prior parallel-decoding algorithms by 1.1–3.2x ¹¹⁵, and nearest-neighbor retrieval-based drafting (NEST) reports 1.8x speedup with quality and attribution gains ¹¹⁴. A comprehensive survey of the family formalizes drafter selection and verification strategies ¹¹¹.

Two boundary findings matter for deployment. First, speculation’s benefit shrinks as batch size grows — the explicit synergy study shows optimal speculation length decreases with batch size, and adaptive strategies add about 9% latency reduction on time-varying loads ¹²²; at high batch, the target model’s own batched compute dominates, so speculative decoding is primarily a low-batch/low-latency technique. Second, the lossless guarantee is about distribution, not quality: retrieval-based drafters report quality gains from the retrieval itself ¹¹⁴, while drafter quality decides the speedup ceiling. SpecExec pushes the family to its limit, generating up to 20 tokens per target iteration to run 50B+ models on consumer GPUs with RAM offloading at 4–6 tokens/s — a 10–18x speedup over sequential offloaded inference ¹¹³.

Theme 5 — Prefill-decode disaggregation and memory pooling

Prefill (prompt processing) is compute-bound and latency-critical; decode (token generation) is memory-bound and throughput-critical. Colocated, they interfere: measurements show naive mixing can cause 10x prefill slowdowns, 5x prefill-decode slowdowns, and 16% decode throughput loss ⁸¹. Disaggregation runs them on separate machines, each configured for its phase. DistServe’s phase-specific parallelism and placement reports serving 7.4x more requests or 12.6x tighter SLOs than prior systems while keeping over 90% of requests within latency bounds ⁴; Splitwise’s phase splitting reports 1.4x throughput at 20% lower cost, or 2.35x under equal power and cost budgets ⁵; Mooncake’s KVCache-centric architecture with tiered CPU/DRAM/SSD cache reports up to 525% simulated throughput gains under SLOs and 75% more requests handled by Kimi on real workloads ⁹; MemServe’s shared memory pool reports up to 42% better job completion time over PD-colocated serving, with a further ~29% from context caching ¹⁰. TetriInfer reports 38% fewer resources with 97% lower average TTFT and 47% lower JCT ⁸¹. Later work refines the binary split: TaiChi unifies aggregation and disaggregation with latency shifting, improving goodput up to 77% under balanced SLOs ⁸³; DuetServe partitions SMs within a single GPU for up to 1.3x throughput ⁸⁴; TokenScale’s token-velocity predictive autoscaling lifts SLO attainment from 50–88% to 80–96% at 4–14% lower cost ⁸⁶; LAPS’s length-aware prefill reduces prefill latency over 30% versus vanilla SGLang under disaggregation ⁸⁸; and dynamic routing of prefill-capable decoders cuts turn-2+ TTFT by ~68% on average ⁸⁷.

The hidden cost is the KV cache transfer between phases. KVDirect shows per-request latency can be cut 55% by pull-based, tensor-centric transfer, while naive message-passing hurts ⁹³; SpectrumKV shows transfer precision matters — at a 50% transfer budget, per-token mixed FP16/INT8/INT4 keeps perplexity change at +1.97% versus +25.85% for a naive baseline, with 50–62% TTFT reduction ⁹⁴; LMCache’s distributed caching layer reports up to 15x throughput combined with vLLM ⁷⁰. Simulation work adds a caution: disaggregation raises serving throughput by up to 75% on current GPUs, but the split’s value depends on workload mix and hardware generation ⁸² — an emerging literature asking “when does disaggregation pay?” rather than assuming it always does. DéjàVu extends the theme toward fault tolerance with KV-cache streaming and state replication, addressing pipeline bubbles where prompt and token latency differ by up to two orders of magnitude ⁹¹.

Theme 6 — Parallelism for long context

Beyond a few hundred thousand tokens, no single GPU holds the KV cache, and parallelism must move from the batch/weights dimension to the sequence dimension. Ring Attention distributes sequences across devices with blockwise attention and fully overlapped KV-block communication ⁶⁰; Context Parallelism reports near-linear prefill scaling to 128 H100s — a 1M-token prefill of Llama-3-405B in 77 seconds at 93% parallelization efficiency ⁶¹. Unified sequence parallelism reports 47% MFU at 208K sequence length on two 8x-A800 nodes ¹²⁰. Infinite-LLM disaggregates attention layers and pools memory cluster-wide for 1.35–3.4x gains on contexts up to 2,000K tokens ⁷³. The serving-side answer is elasticity: LoongServe’s elastic sequence parallelism adjusts parallelism per request and phase, reporting 3.85x over chunked prefill and 5.81x over prefill-decode disaggregation ⁶; Tetris’s chunkwise dynamic sequence parallelism reports up to 4.35x lower TTFT at max sustainable load with 45% more request capacity ⁸⁵. At the other end of the spectrum, HPipe pipelines on the token dimension across heterogeneous commodity devices with up to 2.28x speedup — long context without a cluster ¹¹⁹.

Theme 7 — Mixture-of-experts and inference-cost architectures

MoE models cut inference FLOPs by activating a fraction of parameters per token, and the evidence shows the trade is favorable enough to dominate frontier serving: Mixtral 8x7B activates 13B of 47B parameters while matching or beating Llama-2-70B and GPT-3.5 ⁵⁷; DeepSeek-V2 activates 21B of 236B, cuts KV cache 93.3% via multi-head latent attention, and reports 5.76x maximum generation throughput over its dense predecessor — over 50K generation tokens/s and 100K prompt tokens/s on one 8x-H800 node ⁵⁸; DeepSeek-V3 (671B total, 37B active) trains for 2.788M H800 GPU hours and became the reference for cost-efficient frontier serving ⁵⁹. The systems work behind these: GShard scaled a multilingual MoE Transformer beyond 600B parameters, training on 2048 TPU v3s in 4 days ⁵³; Switch’s simplified top-1 routing gave up to 7x pre-training speedups ⁵⁴; Tutel’s adaptive parallelism reports 4.96x–5.75x MoE-layer speedups ⁵⁵; DeepSeekMoE’s fine-grained expert specialization matches a 67B dense model with 28.5% of the compute ¹¹⁷; GLaM’s sparse activation uses about half the inference FLOPs per token of GPT-3 with better average performance ¹²¹. At serving time, MoE’s problem is memory and all-to-all communication: DeepSpeed-MoE reports 7.3x better latency/cost than prior MoE inference and serves up to 4.5x faster and 9x cheaper than quality-equivalent dense models ⁵⁶; expert offloading with LRU caching runs Mixtral-8x7B interactively on consumer GPUs at 2–3 tokens/s ¹⁰²; ExFlow’s context-coherent expert placement replaces two all-to-alls with one, improving throughput up to 2.2x over DeepSpeed-MoE ¹¹⁸.

Architecture changes with explicit inference motivation form the supporting cast. Multi-query attention cut incremental decoder step time from 47ms to 3.9ms per step — about 12x — by sharing one key/value head ⁶²; grouped-query attention recovers most of the quality loss at MQA’s speed, with an uptraining recipe costing only 5% of pre-training compute ⁶³; multi-head latent attention compresses the cache into a low-rank latent, which is where DeepSeek-V2’s 93.3% KV reduction comes from ⁵⁸. Mamba’s selective state spaces report 5x higher generation throughput than same-size transformers with linear scaling in sequence length ⁶⁴. These architecture results are included because they define the efficiency frontier the serving systems are built to exploit.

Theme 8 — Memory-limited deployment: offloading and consumer hardware

When no GPU big enough exists, the alternatives are offloading to CPU memory or splitting across weak devices. FlexGen pioneered throughput-oriented offloading, reporting 1 token/s for OPT-175B on a single 16GB GPU — up to 100x the maximum throughput of prior offloading systems — via effective batching with block-wise weight streaming ¹¹. PowerInfer’s hot/cold neuron split (frequent neurons on GPU, rare on CPU) reports up to 11.69x over llama.cpp on a single RTX 4090, with OPT-30B reaching 82% of an A100’s generation rate ⁶⁵. “LLM in a flash” showed models up to twice the size of available DRAM can run with 4–5x (CPU) and 20–25x (GPU) speedups over naive loading via windowing and row-column bundling ¹⁰⁵. LLMCad reports 9.3x faster token generation on mobile devices, where weight reloading alone can lengthen latency by 59–224x ¹⁰⁴. SpecExec ¹¹³ and the MoE-offloading work ¹⁰² extend the same story to speculative decoding and experts respectively; fine-grained CPU-GPU allocation adds 105% on commodity hardware ¹¹⁶. The consistent finding across all of them: offloading converts the problem from memory-capacity-bound to memory-bandwidth-bound, and the achievable rate tracks DRAM bandwidth — 2–6 tokens/s for 50B+ models — which is why quantization (fewer bytes to move) multiplies rather than merely adds.

Where the evidence disagrees

Lossless claims vs measured degradation. Speculative decoding’s “provably lossless” guarantee is exact — it preserves the target distribution by construction ⁴²⁴⁶. KV-cache eviction and compression papers, by contrast, routinely claim “negligible” or “no” quality loss, yet the independent KV-cache benchmark shows degradation that grows with compression ratio and depends on task ¹⁰⁹, and SpectrumKV’s controlled comparison shows the same ⁹⁴. The apparent disagreement dissolves once measures are separated: perplexity is insensitive to cache loss; retrieval and long-context tasks are not. Papers reporting only perplexity are measuring the wrong thing.

The size of speculation’s gain. Reported speedups span 1.5x to 6.5x for the same underlying idea. The resolution is batch size and drafter quality: at batch 1 with a well-matched drafter, 3–6.5x is reproducible ⁷⁶; at production batch sizes the gain collapses because the target model’s batched forward pass dominates, and the explicit batching-synergy study documents the curve ¹²². “Speculative decoding gives 2–3x” and “speculative decoding barely helps in production” are both true statements about different operating points.

Quantization quality at the frontier. Whether 2-bit is “viable” depends on the yardstick: QuIP/QuIP# argue viability on perplexity and downstream tasks ³¹³²; SpinQuant’s own numbers show 4-bit still beats 2-bit by wide margins ³³; the k-bit scaling-law study says 4-bit is the sweet spot ⁴⁰; BitNet argues 1-bit can be trained, not just post-quantized, to competitive quality ⁴¹. The camps are not really contradicting — they are answering different questions (post-training vs trained-in quantization, perplexity vs task accuracy) — but a reader will find both “2-bit works” and “2-bit doesn’t work” claims in this literature.

Disaggregation’s universality. DistServe, Splitwise, and Mooncake show large gains ⁴⁵⁹; the simulation literature and the KV-transfer papers show the gains depend on workload mix, hardware generation, and the cost of moving KV state ⁸²⁹³⁹⁴. The field is moving from “disaggregate everything” to “disaggregate when the phase ratio and transfer cost justify it” — the disagreement is temporal, and the later position is better supported.

Where the throughput comes from. FastServe-style scheduling attributes gains to queueing discipline ¹³; vLLM and Sarathi attribute theirs to batching and memory efficiency ¹². These are compatible mechanisms, but the field’s habit of benchmarking against vLLM as the universal baseline means every system paper reports “x-times over vLLM” without decomposing which component of vLLM it beat — a structural weakness in the literature’s comparability, not a resolved disagreement.

Gaps and open questions

No shared benchmark. The single most important gap: there is no standard harness for LLM serving — no canonical hardware, workload mix, and metric set — so cross-paper comparison is impossible. Vidur ⁷⁷ and BurstGPT ⁷⁸ are partial answers (simulation and workload data), and the KV-cache and quantization benchmarks ¹⁰⁹¹¹⁰ are the first domain-level attempts. What would settle it: a community benchmark for inference serving with published per-technique ablations on identical hardware.

Composition of techniques. Nearly every paper optimizes one layer in isolation, yet production systems stack batching, KV compression, quantization, speculation, and disaggregation. The interaction effects are known to be non-additive in at least one direction — speculation shrinks with batching ¹²² — but no included source systematically maps the joint space; the field needs a composition study.

Quality measurement. KV eviction and compression work needs a standard long-context quality suite that includes retrieval and multi-hop reasoning, not just perplexity; the benchmark evidence shows today’s claims are optimistic ¹⁰⁹.

Production economics. Reported metrics are almost all latency and throughput; cost per token and SLO attainment under real production traces appear in only a handful of sources ⁷⁸⁸⁶⁹. The infrastructure question — what does a token actually cost at the fleet level, and which optimization moves that number most — is under-measured in the literature.

Systematic long-context serving. Million-token serving exists on different axes — quantized cache ²⁰, selective heads ⁹⁷, cluster parallelism ⁶¹ — but a system combining all three with production-grade scheduling has not been demonstrated in the retrieved literature.

Confidence and limitations

Confidence is moderate. The mechanisms are physically grounded and replicated: inference is memory-bound ⁷⁹, batching and KV-cache management move the needle by multiples, quantization to 4-bit is near-free, speculation is lossless and 2–3.5x at low batch, disaggregation pays when phase profiles are skewed. The strongest evidence is in serving systems and quantization, where multiple independent groups converge on similar numbers. The weakest is quality preservation at aggressive KV compression and sub-4-bit quantization, where self-reported perplexity hides task-level degradation ¹⁰⁹, and the whole field’s “up to” culture means magnitudes should be read as ceilings, not expectations.

This review’s own limitations: 28 of 124 sources were abstract-only (mostly ACL-anthology papers whose full texts were not openly retrievable); Orca (continuous batching’s origin) and TwinPilots are absent because they have no DOI or reachable abstract; the corpus is English-language and arXiv-heavy, which overweights preprint claims; the date cutoff (2026-08-08) excludes anything newer; and several load-bearing numbers are the authors’ own reports on their own hardware, verified present in the text but not independently reproduced. Numbers flagged “up to” should be treated as such.

Jump to references ↓

Evidence table

key	design	sample	measure	finding	limitations	confidence	access	note
kwon2023efficient	system	OPT, LLaMA, and other popular LLMs; NVIDIA GPUs; continuous batching with paged KV cache	serving throughput at matched latency	vLLM with PagedAttention achieves near-zero KV-cache waste and improves throughput of popular LLMs by 2-4x at the same latency level compared to state-of-the-art systems such as FasterTransformer and Orca.	Abstract-only access locally; improvements reported as larger for longer sequences, bigger models, and complex decoding; original eval predates later systems.	moderate	abstract-only	PagedAttention/vLLM is the reference point for KV-cache memory management and continuous batching in nearly all later serving work.
agrawal2023sarathi	system	LLaMA-13B (A6000), LLaMA-33B (A100), GPT-3 with 8-way pipeline + 8-way tensor parallelism on simulated 64-A100 cluster	decode throughput, end-to-end throughput, pipeline bubble ratio	Sarathi's chunked-prefills and decode-maximal batching improve decode throughput by up to 10x (LLaMA-13B/A6000) and 4.25x (LLaMA-33B/A100) with 1.33x/1.25x end-to-end gains, and reduce pipeline bubbles by 6.29x for GPT-3 for a 1.91x end-to-end throughput improvement.	Decode piggyback gains depend on prefill/decode batch composition; GPT-3 pipeline results use a simulated cluster; decode per-token cost drops from 12.49 ms to 1.2 ms only when prefills saturate the batch.	high	full-text	Origin of chunked-prefill + decode-maximal batching now standard in vLLM-class serving engines.
agrawal2024taming	system	Mistral-7B (1x A100), Yi-34B (2x A100), Falcon-180B (pipeline parallel); arxiv-summarisation and other traces	serving capacity under tail-latency SLOs	Sarathi-Serve's chunked-prefills with stall-free batching deliver 2.6x higher serving capacity for Mistral-7B on one A100, up to 3.7x for Yi-34B on two A100s, and up to 5.6x end-to-end with pipeline parallelism on Falcon-180B versus vLLM.	Gains depend on tail-latency SLOs and workload traces; chunk sizing adds scheduling complexity; comparisons are vs vLLM only.	high	full-text	Introduces chunked prefill + stall-free batching, the scheduling recipe adopted by vLLM and most modern serving engines.
zhong2024distserve	system	13B+ LLMs, synthetic and real workloads (512-token input, 64-token output etc.); NVIDIA 80GB A100 GPUs; TTFT and TPOT SLOs	per-GPU goodput (max request rate within TTFT+TPOT SLOs)	DistServe's prefill/decode disaggregation with phase-specific parallelism and placement serves 7.4x more requests or 12.6x tighter SLOs than state-of-the-art systems while keeping over 90% of requests within latency constraints.	Requires cluster bandwidth-aware placement; disaggregation adds KV transfer overhead between phases; benefits depend on workload mix.	high	full-text	Foundational PD-disaggregation system; defines per-phase SLO (TTFT/TPOT) as the serving objective.
patel2023splitwise	system	LLM inference clusters; homogeneous and heterogeneous (phase-specialized) machines; fast backplane interconnects	Throughput, cost, power of inference clusters	Splitting compute-intensive prefill and memory-intensive decode onto separate machines yields up to 1.4x higher throughput at 20% lower cost, or 2.35x more throughput under the same cost and power budgets.	Requires fast GPU-cluster interconnects for state transfer; workload-dependent phase ratios	high	full-text	Foundational evidence for phase-disaggregation: decode does not need latest-GPU compute, enabling cheaper, power-efficient clusters.
wu2024loongserve	system	Real-world serving datasets with highly variable request lengths and phase-wise resource usage	throughput	LoongServe's elastic sequence parallelism (ESP) improves throughput by up to 3.85x compared to chunked prefill and 5.81x compared to prefill-decoding disaggregation, by elastically adjusting parallelism degree per request/phase and cutting KV migration overhead.	Abstract-only record (numbers from the abstract); benefits depend on workload variance and ESP-capable execution stack.	high	abstract-only	Argues static parallelism strategies waste resources under variable-length requests — dynamic parallelism for serving.
duan2024muxserve	system	Multiple LLM endpoints of varying popularity (e.g., LLaMA-7B) on a 32-GPU cluster; synthetic and real workloads	throughput, requests within 99% SLO	MuxServe's spatial-temporal multiplexing (CUDA-MPS SM partitioning plus popularity-aware colocation) achieves up to 1.8x higher throughput or processes 2.9x more requests within 99% SLO attainment versus prior systems.	Needs workload popularity estimates; CUDA MPS partitioning granularity constrains colocation; evaluated mainly on LLaMA-7B-scale models.	high	full-text	Multi-model serving via phase-aware colocation — relevant to heterogeneous endpoint fleets.
zheng2024sglang	system	Llama-7B, Mixtral-8x7B, Llama-2-70B, LLaVA-Next-34B; agent control, JSON decoding, RAG, multi-turn chat workloads	end-to-end throughput (programs completed per second)	SGLang with RadixAttention KV-cache reuse and compressed FSM decoding achieves up to 6.4x higher throughput than state-of-the-art inference systems across LLM and multimodal workloads.	Gains depend on prompt prefix sharing across calls; RadixAttention adds scheduling complexity and was only partially integrated into vLLM at the time.	high	full-text	Canonical system evidence that structured KV cache reuse (radix tree, LRU eviction) accelerates multi-call LLM programs.
qin2024mooncake	system	Kimi production platform (Moonshot AI); long-context workloads where input tokens are 10-100x output tokens; SLOs TTFT_P90=10x, TBT_P90=5x	effective throughput, request capacity, TTFT/TBT SLO attainment	Mooncake's KVCache-centric disaggregated prefill/decode architecture with tiered CPU/DRAM/SSD cache achieves up to 525% higher throughput than baseline under SLOs in simulated long-context scenarios and lets Kimi handle 75% more requests on real workloads.	525% figure comes from simulated scenarios; real-workload result is increased request capacity rather than latency gain; cache policy analysis based on a subset trace (hit ratio 30%->50% from 1,000 to 50,000 cache blocks).	high	full-text	Flagship production evidence for KV-cache-centric disaggregation and early-rejection scheduling.
hu2024memserve	system	ShareGPT and LooGLE workloads; vLLM-based PD-colocated vs PD-disaggregated vs disaggregated+caching setups	Job completion time (JCT); TTFT; cache hit rate	MemServe's MemPool-based disaggregated inference improves JCT by up to 42% over PD-colocated serving (ShareGPT), and combining it with context caching yields a further ~29% JCT gain.	Single-node disaggregation focus; numbers depend on cache-reuse patterns of the workload	high	full-text	Shows the compounding benefit of unifying context caching with disaggregated inference via an elastic distributed memory pool.
sheng2023flexgen	system	OPT-175B and OPT-30B on a single 16GB GPU (T4) with CPU DRAM and SSD; batch up to 144	generation throughput (tokens/s), latency-throughput Pareto frontier	FlexGen reaches 1 token/s generation throughput for OPT-175B on a single 16GB GPU with an effective batch size of 144, up to 100x higher maximum throughput than prior offloading systems, using 4-bit weight and attention-cache compression.	Optimized for latency-insensitive batch throughput, not interactive latency; relies on CPU/disk offloading and 4-bit compression with small accuracy loss.	high	full-text	Shows LLM-scale serving on commodity single-GPU hardware via linear-programming-guided tensor placement and offloading.
pope2022efficiently	benchmark	PaLM 540B and 500B+ models; TPU v4 slices; 2048-token context; int8 weight quantization; low and large batch sizes	per-token latency; model FLOPs utilization (MFU)	With an analytical partitioning model plus low-level optimizations, the authors reach 29ms per token at low batch size (int8) and 76% MFU at large batch on PaLM 540B, and show multiquery attention's lower memory enables up to 32x larger context lengths.	TPU-specific (TPU v4); analytical model assumes known hardware costs; FasterTransformer comparison predates later optimizations.	high	full-text	Establishes the latency-vs-MFU Pareto framing and the KV-memory/context-length tradeoff for giant models.
wu2023fast	system	distributed LLM serving (FastServe prototype); real-world workload traces; GPU memory offload of intermediate states	throughput at matched average/tail latency	FastServe's skip-join MLFQ preemptive scheduler improves throughput by up to 31.4x and 17.9x versus vLLM under the same average and tail latency requirements, respectively, exploiting that up to 90% of end-to-end latency is queueing delay.	Assumes input-length information for initial queue assignment; token-level preemption and state offloading add overhead; numbers from the authors' prototype.	high	full-text	First preemptive (token-granularity) LLM serving scheduler targeting head-of-line blocking in interactive inference.
sheng2023s	system	Many LoRA adapters over one base model; single and multiple GPUs; ranks and KV-cache tensors managed via unified paging	Serving throughput; number of concurrently served adapters	S-LoRA's unified paging and heterogeneous-batching kernels serve thousands of LoRA adapters on a single GPU and improve throughput by up to 4x over HuggingFace PEFT and vLLM with naive LoRA support.	Assumes adapter weights fit in main memory; custom kernels needed for heterogeneous ranks	high	full-text	Shows memory-pooling (unified paging) generalizes from KV cache to adapter weights for multi-tenant fine-tuned serving.
zhang2023h	system	OPT-6.7B/30B, LLaMA, GPT-NeoX; single NVIDIA A100 80GB; lm-eval-harness and HELM tasks	throughput vs baseline systems; latency; KV cache size	H2O with 20% heavy-hitter budget improves throughput up to 29x, 29x, and 3x over DeepSpeed Zero-Inference, Hugging Face Accelerate, and FlexGen respectively, and cuts latency up to 1.9x at the same batch size, exploiting >95% attention sparsity.	Heavy-hitter retention biases toward frequent tokens; eviction may hurt tasks needing rare distant context; measured against 2023-era baselines.	high	full-text	Landmark KV eviction paper; introduces heavy-hitter oracle and submodular formulation of eviction.
liu2023scissorhands	system	OPT and other LLMs; batch size 128, sequence length 2048; fixed-budget KV cache	KV cache memory usage; compression ratio	Scissorhands, based on the persistence-of-importance hypothesis (attention overlap over 90% in most layers), reduces KV cache memory up to 5x without quality loss, and up to 20x when combined with 4-bit quantization.	Eviction policy assumes persistent importance; very long-context degradation not fully covered; needs tuning of budget thresholds.	high	full-text	Early training-free KV eviction evidence linking attention persistence to cache budget reduction.
li2024snapkv	system	LWM-Text-Chat-1M, LongChat-7B, Mistral-7B, Mixtral-8x7B, Command-R; A100-80GB; 16K-380K token inputs	generation speed, memory efficiency, max context length, decoding latency	SnapKV delivers a 3.6x increase in generation speed and 8.2x memory-efficiency gain at 16K-token inputs (decoding <40ms/token vs >100ms baseline), and processes up to 380K context tokens on a single A100-80GB where the baseline OOMs at 33K, at up to 380x KV compression.	Assumes prompt importance is concentrated and discoverable in an observation window at the prompt end; needs per-head position selection and pooling tuning.	high	full-text	Training-free prompt KV compression by per-head important-position selection; strong for long-context memory and latency reduction.
xiao2023efficient	system	Llama-2-7B/13B/70B, MPT-7B/30B, Falcon-7B/40B, Pythia; single A6000; streaming up to 4M tokens	per-token decoding latency, memory usage, max streamable length	StreamingLLM keeps 4 attention-sink tokens plus a sliding window to let models trained on finite windows stream up to 4M tokens, outperforming the sliding-window-with-recomputation baseline by up to 22.2x in per-token decoding speed.	Authors show accuracy drops vs truncation baseline on LongBench when the cache is too small; relies on the attention-sink phenomenon holding for the model.	high	full-text	Attention-sink insight underpins KV eviction policies (sink+recent) used across streaming, drafting, and cache-compression systems.
liu2023kivi	system	Llama, Falcon, Mistral models; real LLM inference workloads; 2-bit KV cache (per-channel keys, per-token values)	Peak memory, achievable batch size, throughput, generation quality	KIVI's tuning-free asymmetric 2-bit KV quantization uses 2.6x less peak memory (including weights), enabling up to 4x larger batch sizes and 2.35x-3.47x throughput while maintaining near-original quality.	KV-cache-only compression; per-channel key quantization needs padding logic; quality at very long contexts not guaranteed	high	full-text	Landmark evidence that KV cache tolerates 2-bit quantization with per-channel/per-token asymmetry, directly attacking the decode memory bottleneck.
hooper2024kvquant	benchmark	LLaMA, Llama-2, Llama-3, Mistral; A100-80GB; Wikitext-2 and C4	perplexity degradation, achievable context length, kernel speedup	KVQuant achieves <0.1 perplexity degradation with 3-bit KV cache quantization, enabling LLaMA-7B serving at up to 1M context on a single A100-80GB (10M on an 8-GPU system) and up to ~1.7x speedup over fp16 matrix-vector multiply in custom CUDA kernels.	Evaluated on perplexity for two datasets; requires custom kernels; per-layer sensitivity tuning and non-uniform datatypes add engineering complexity.	high	full-text	Sub-4-bit KV cache quantization suite (per-channel, pre-RoPE, non-uniform, dense-and-sparse) that makes million-token contexts feasible on one GPU.
kang2024gear	system	LLaMA-2 and other LLMs; generative tasks including math and chain-of-thought; batch 128, 1024-token input context	KV cache compression ratio; throughput; peak memory	GEAR (quantization + low-rank + sparse error correction) achieves near-lossless 4-bit KV compression with up to 2.38x throughput improvement and 2.29x peak-memory reduction (full-text: 2.10x-5.07x throughput, 2.39x memory; 14.95% accuracy gain over best baseline at 2-bit).	Adds low-rank/sparse overhead kernels; error-correction components complicate deployment; gains largest at high compression ratios.	high	full-text	Shows hybrid quantization+low-rank+sparse beats uniform quantization/eviction for high-ratio KV compression.
gim2023prompt	system	Llama2, Falcon, MPT; LongBench document QA and recommendation prompts; CPU and GPU memory stores	TTFT latency reduction	Prompt Cache's attention-state reuse for schema-defined prompt modules reduces TTFT by 8x (GPU) to 60x (CPU) for long prompts, with 1.5x-10x reported in the full-text GPU evaluation, while preserving output accuracy.	Requires prompts to contain reusable schema-declared segments; accuracy of positional reuse depends on schema correctness; CPU numbers reflect memory-hierarchy effects.	high	full-text	Early evidence that cross-request KV reuse (precomputed attention states) accelerates prompt-heavy workloads.
yao2024cacheblend	system	Three open-source LLMs (incl. Llama-7B, Llama-70B); four benchmark datasets (QA, summarization); ~4K-token contexts; NVMe/CPU-RAM KV storage	TTFT, inference throughput, F1/Rouge-L	CacheBlend reduces TTFT by 2.2-3.3x and increases inference throughput by 2.8-5x versus full KV recompute without compromising generation quality, by reusing precomputed KV caches of non-prefix chunks and selectively recomputing KV for ~15% of high-deviation tokens (e.g., 3 ms recompute per layer vs 16 ms NVMe load for Llama-7B).	Requires precomputed caches and storage tiering; 15% recompute ratio is a heuristic; KV-loading must hide recompute delay on fast devices.	high	full-text	Enables RAG-style chunk reuse beyond prefix caching, a key building block for cache-centric serving.
jin2025ragcache	system	RAG workloads; open-source and production datasets; vLLM + Faiss baseline	Time to first token (TTFT); end-to-end throughput	RAGCache organizes retrieved-knowledge KV states in a knowledge tree across GPU/host memory with dynamic speculative pipelining, reducing TTFT by up to 4x and improving throughput by up to 2.1x versus vLLM integrated with Faiss.	Abstract-only record; gains specific to RAG-shaped workloads with reusable retrieved passages	high	abstract-only	Shows KV caching across the memory hierarchy plus retrieval-generation overlap as levers for RAG serving latency.
liu2024cachegen	system	Popular LLMs and datasets; KV cache reuse across inputs with network transfer; bandwidth-adaptive compression levels	KV cache bitstream size and total context fetch+process delay	CacheGen reduces KV cache size by 3.5-4.3x and the total delay of fetching and processing contexts by 3.2-3.7x compared to recent KV-reuse systems, with negligible impact on response quality.	Gains assume KV-cache reuse across requests and variable network bandwidth; quality impact is small but nonzero under aggressive compression.	high	abstract-only	Lossy-but-tuned KV cache encoding that attacks context-loading network delay in long-context serving.
frantar2022gptq	system	GPT/OPT models up to 175B; 3-4 bit one-shot quantization; NVIDIA A100 and A6000	Quantization time, bitwidth vs accuracy, end-to-end inference speedup vs FP16	GPTQ quantizes 175B-parameter GPT models in ~4 GPU hours to 3-4 bits per weight with negligible accuracy loss, enabling single-GPU 175B inference with ~3.25x (A100) and ~4.5x (A6000) end-to-end speedups over FP16.	Weight-only; accuracy degrades in 2-bit/ternary extreme regime; speedups measured on generation-bound workloads	high	full-text	Foundational one-shot PTQ result establishing 4-bit weights as the practical default for LLM deployment.
lin2023awq	system	LLaMA/Llama-2 up to 70B, instruction-tuned and multi-modal LMs; desktop and mobile GPUs (e.g., Jetson Orin Nano 8GB)	inference speedup vs FP16, deployment feasibility	AWQ's activation-aware scaling of 1% salient weight channels, implemented in TinyChat, yields more than 3x (3-4x) speedup over HuggingFace FP16 on desktop and mobile GPUs, putting a 4-bit 70B Llama-2 on mobile hardware.	Weight-only 4-bit quantization; needs offline activation statistics; TinyChat kernel optimizations are platform-specific.	high	full-text	Hardware-friendly weight-only PTQ that became a standard on-device 4-bit serving stack; complements KV-cache and activation quantization.
xiao2022smoothquant	system	OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, Mixtral; W8A8 INT8	inference speedup, memory reduction, model size servable per node	SmoothQuant migrates activation quantization difficulty into weights to enable W8A8 INT8 for all matmuls, delivering up to 1.56x speedup and 2x memory reduction with negligible accuracy loss, and serving a 530B-parameter LLM within a single node.	Limited to 8-bit (not sub-4-bit) quantization; accuracy can degrade on outlier-heavy activations; needs offline smoothing statistics per model.	high	full-text	Foundational W8A8 PTQ that halves memory and nearly doubles GEMM throughput; enabler of single-node 500B+ serving.
dettmers2022llm	system	Transformers up to 175B parameters (OPT-175B, BLOOM); feed-forward and attention projection layers quantized to Int8	GPU memory for inference and accuracy retention	LLM.int8() cuts inference memory by half while retaining full-precision performance, enabling 175B-parameter models to run on a single server with consumer GPUs with more than 99.9% of values computed in 8-bit.	Mixed-precision decomposition still routes outlier dimensions through 16-bit matmul, so speedup depends on outlier fraction; memory halving is the headline, not raw latency.	high	abstract-only	Foundational outlier-aware 8-bit quantization that made large-model inference accessible on commodity hardware.
yao2022zeroquant	system	BERT and GPT3-style models, GPT-J6B, GPT-NeoX20B; INT8 weights+activations, INT4 FC weights; layer-by-layer distillation (LKD)	Inference speedup vs FP16; memory footprint reduction; accuracy	ZeroQuant's end-to-end INT8 quantization with optimized backends achieves up to 5.19x/4.16x speedups over FP16 on BERT/GPT3-style models, and INT4+INT8 gives 3x memory reduction with up to 5.2x efficiency on GPT-J6B/GPT-NeoX20B.	INT8 activation quantization needs hardware with good INT8 support; LKD adds distillation complexity	high	full-text	Early full-stack W8A8 evidence, including a data-free distillation trick, relevant to activation-quantization history.
chee2023quip	theoretical	LLM-scale weight quantization (2 bits per weight); incoherence processing with random orthogonal transforms; OPTQ comparison	Quantization quality (perplexity) at 2-bit weights; theoretical guarantees	QuIP's incoherence-processing plus adaptive rounding yields the first LLM quantization methods with viable results at 2 bits per weight, backed by the first theoretical analysis for an LLM-scale quantization algorithm.	Weight-only; theory covers quadratic-rounding proxy, not end-to-end serving performance	moderate	full-text	Theoretical anchor for why incoherence enables extreme low-bit weight quantization, informing later QuIP# and 2-bit methods.
tseng2024quip	system	LLMs at <=4 bits per weight; randomized Hadamard incoherence; E8-lattice vector-quantization codebooks; fine-tuning	Perplexity/quality at extreme bitwidths; codebook efficiency	QuIP# combines Hadamard incoherence processing, E8-lattice vector quantization, and fine-tuning to reach state-of-the-art post-training quantization quality at extreme compression (<=4 bits per weight, including viable 2-bit regimes).	Abstract reports no single magnitude (e.g., speedup) number; weight-only, no KV/activation quantization	moderate	full-text	Advances the 2-bit frontier for weight-only PTQ via lattice codebooks; complements GPTQ-style methods in the review.
liu2024spinquant	system	LLaMA-2 7B, LLaMA-3 8B; zero-shot reasoning tasks; 4-bit weight+activation+KV quantization	accuracy gap to full precision (zero-shot reasoning)	SpinQuant's learned rotation matrices narrow the 4-bit W4A4KV4 accuracy gap to 2.9 points on LLaMA-2 7B (beating LLM-QAT by 19.1 and SmoothQuant by 25.0 points), and cut the gap by up to 45.1% relative to QuaRot on LLaMA-3 8B.	Rotation optimization requires calibration data and training of rotation matrices; random rotations vary by up to 13 points, so initialization matters.	high	full-text	Evidence that learned (not random) rotation matrices are key to accurate full-4-bit quantization including KV cache.
micikevicius2022fp8	benchmark	CNNs, RNNs, Transformer-based models incl. GPT up to 175B parameters; Wikipedia and The Pile corpora	training loss/perplexity, PTQ accuracy	Proposes FP8 E4M3/E5M2 formats whose training results match 16-bit sessions on models up to 175B parameters and whose post-training quantization preserves accuracy where int8 PTQ fails (BERT SQuAD), though FP8-casting residuals without per-tensor scaling degrades perplexity (12.59 vs 10.19 bfloat16 baseline).	Format-level study without inference-speed measurements; per-tensor scaling needed for some tensors; FP8 hardware availability was limited at publication.	high	full-text	Defines the FP8 standard that underpins modern 8-bit inference engines — foundational for the quantization chapter.
lin2024qserve	system	Llama-3-8B, Qwen1.5-72B; A100 and L40S GPUs; W4A8KV4 (QoQ) quantization; TensorRT-LLM baseline	Maximum achievable serving throughput; dollar cost of serving	QServe's W4A8KV4 QoQ quantization with low-dequantization kernels improves max serving throughput by 1.2x (A100) / 1.4x (L40S) for Llama-3-8B and 2.4x (A100) / 3.5x (L40S) for Qwen1.5-72B over TensorRT-LLM, cutting serving dollar cost by 3x.	Targets cloud GPUs with INT4 tensor-core support; 4-bit KV quantization needed SmoothAttention to preserve accuracy	high	full-text	Key evidence that W4A8KV4 with dequantization-aware kernels unlocks INT4 speedups in large-batch cloud serving, not just edge.
zhao2023atom	system	Llama models; 4-bit weight-activation (W4A4) quantization; NVIDIA A100 serving with INT4 operators; WikiText2 perplexity check	end-to-end serving throughput (tokens/s) at fixed latency	Atom's mixed-precision fine-grained 4-bit quantization improves end-to-end serving throughput by up to 7.7x over FP16, 5.5x over W4A16, and 2.5x over INT8 (W8A8) at the same latency target with near-FP16 perplexity.	Requires INT4-capable GPU operators (Ampere+); outlier handling adds complexity; accuracy depends on mixed-precision channel splitting.	high	full-text	Shows low-bit weight-activation quantization translating directly into serving throughput via INT4 tensor cores.
shao2023omniquant	benchmark	LLaMA-2 family 7-70B, single A100-40G, 128 calibration samples; W4A4/W6A6/W4A16/W3A16/W2A16	quantization accuracy across bit-widths, PTQ time and data efficiency	OmniQuant's learnable weight clipping and equivalent transformation let the LLaMA-2 7-70B family be quantized on a single A100-40G GPU within 1-16 hours using 128 samples, with strong results down to W4A4 and W2A16.	Abstract/fulltext report accuracy and cost but no absolute latency/throughput speedup numbers; block-wise optimization needs calibration data.	high	full-text	Omnidirectional PTQ covering extreme low-bit weight-activation regimes at QAT-like quality with PTQ-level cost.
kim2023squeezellm	benchmark	LLaMA-7B/13B/30B/65B; A6000 GPU; C4 and WikiText benchmarks; 3-4 bit weight-only quantization	perplexity, latency speedup	SqueezeLLM's sensitivity-based non-uniform 3-bit quantization reduces the perplexity gap to the FP16 baseline by up to 2.1x versus state-of-the-art methods at equal memory (LLaMA-7B C4 perplexity 7.75 vs 28.26 for uniform 3-bit) and achieves up to 2.3-2.4x speedup on an A6000.	Weight-only quantization (activations stay 16-bit); speedups demonstrated in the single-batch memory-bound regime that dominates generative inference.	high	full-text	Shows 3-bit weight quantization can be lossless-ish with non-uniform sensitivity-aware schemes.
dettmers2023spqr	benchmark	LLaMA and Falcon families (1-33B), single 24GB consumer GPU	perplexity loss, memory compression ratio, inference speedup	SpQR isolates outlier weights in higher precision and compresses the rest to 3-4 bits, achieving <1% relative perplexity loss while enabling a 33B model on a single 24GB GPU at a 15% speedup with more than 4x memory compression.	Weight-only compression; speedup over fp16 is modest (15%); outlier storage and grouped-scale metadata add format complexity.	high	full-text	Early near-lossless 3-4 bit weight quantization enabling consumer-GPU deployment; baseline for later outlier-aware PTQ methods.
dettmers2022case	theoretical	BLOOM, OPT, NeoX/Pythia, GPT-2 families, 19M-176B parameters; 3-16 bit precision; 35,000+ experiments	zero-shot accuracy per total model bits (bit-level inference scaling laws)	Across 35,000+ experiments, 4-bit precision is almost universally optimal for total model bits versus zero-shot accuracy, with small quantization block size and float data types the only effective improvements.	Zero-shot tasks only; scaling laws derived for weights, not KV cache or activations; applies to 2022-era quantization methods.	high	full-text	Theoretical anchor for the field's 4-bit default in weight quantization.
wang2023bitnet	theoretical	1-bit BitLinear transformers trained from scratch up to ~3B scale; language modeling	memory footprint and energy vs accuracy; scaling law slope	BitNet trains 1-bit-weight Transformers that achieve competitive language modeling performance with substantially reduced memory footprint and energy versus FP16 and 8-bit quantized baselines, while exhibiting a scaling law similar to full-precision Transformers.	No magnitude numbers (percent memory/energy savings) in the available text; requires training from scratch, so not a drop-in PTQ solution.	moderate	full-text	Evidence for 1-bit weights as an end-goal architecture; motivates extreme low-bit inference research.
chen2023accelerating	system	Chinchilla 70B; distributed setup; draft-model parallel scoring; modified rejection sampling	Decoding speedup; sample-quality preservation	Speculative sampling with a small draft model accelerates Chinchilla-70B decoding by 2-2.5x in a distributed setup while provably preserving the target distribution.	Needs a good draft model; speedup bounded by draft acceptance rate and parallel-scoring cost	high	full-text	Foundational speculative-decoding result: parallel verification makes drafting an exact, distribution-preserving speedup.
leviathan2022fast	theoretical	T5-XXL (11B) with T5-small/base/large drafts; 97M GPT-like model; LaMDA 137B; TPU-v4, batch 1	wall-clock decoding speedup at identical output distribution	Speculative decoding gives 2x-3x out-of-the-box wall-clock speedup on T5-XXL versus T5X with identical outputs (up to 3.4x on translation with argmax sampling and a T5-small draft).	Authors flag that it increases arithmetic operations and needs spare compute; speedup depends on draft-model acceptance (alpha) and draft cost.	high	full-text	Foundational speculative-decoding paper: formal rejection sampling guaranteeing the target distribution, plus speedup analysis.
cai2024medusa	system	Vicuna-7B/13B/33B, Zephyr-7B; batch size 1; MT-Bench; single A100	wall-clock decoding speedup, tokens per second	Medusa's multiple parallel decoding heads with tree attention give over 2.2x speedup for Medusa-1 (frozen backbone) and 2.3-3.6x for Medusa-2 (2.18x/2.83x measured on Vicuna-7B, up to 3.62x on extraction tasks) without compromising generation quality.	Requires fine-tuning extra heads (Medusa-2 needs a special recipe); evaluated mainly at batch size 1; speedup drops at higher temperatures and for harder tasks.	high	full-text	Draft-model-free parallel decoding (multi-token heads + tree verification); benchmark anchor for tree-based speculative methods.
li2024eagle	system	Vicuna and LLaMA2-Chat series (incl. 70B), Mixtral 8x7B Instruct; dialogue, code, math, instruction-following tasks	Latency speedup; throughput; distribution preservation	EAGLE's feature-level (second-to-top-layer) autoregression with one-step-ahead token conditioning achieves 2.7x-3.5x latency speedup on LLaMA2-Chat 70B and doubles throughput while preserving the target distribution.	Requires training a small feature-level draft head; speedup varies with task and model family	high	full-text	State-of-the-art draft-based speculation: predicting at the feature level yields higher acceptance than token-level drafts.
li2024eagle2	system	Three LLM series (incl. Vicuna/LLaMA2-Chat); six tasks (MT-bench, HumanEval, GSM8K, etc.); temperature=1	speedup ratio (tokens/s vs autoregressive baseline)	EAGLE-2's context-aware dynamic draft tree achieves 3.05x-4.26x speedups, 20-40% faster than EAGLE-1 (and about 2x faster than Medusa, 2.3x faster than Lookahead on MT-bench), while provably preserving the output distribution.	Draft model must be well-calibrated (EAGLE-style); speedup depends on task acceptance rates; extra tree-scheduling logic required.	high	full-text	State-of-the-art lossless speculative decoding: confidence-guided dynamic draft trees beat static trees.
stern2018blockwise	theoretical	Transformer seq2seq machine translation (WMT-class) and image super-resolution models; block sizes up to 10	decoding iteration reduction, wall-clock speedup, BLEU	Blockwise parallel decoding reduces decoding iterations by up to 2x over greedy decoding with no quality loss (up to 7x iteration reduction and 4x wall-clock speedup with relaxed verification and fine-tuned models, e.g., mean accepted block size 6.79 at block size 10).	Requires parallel-capable architectures and (for large gains) fine-tuning plus approximate acceptance; exact-match verification yields mean accepted block sizes below 1.5 on frozen models.	high	full-text	Theoretical origin of blockwise speculative decoding — canonical citation for the speculative-decoding chapter.
he2024rest	system	7B and 13B language models, single-batch setting, code and text generation	End-to-end decoding speedup via retrieval-based drafting	REST achieves a significant speedup of 1.62x to 2.36x on code or text generation for 7B/13B models in single-batch settings.	Single-batch evaluation only; speedup depends on the retrieval corpus matching generation patterns; draft quality bounded by retrieved tokens.	high	abstract-only	Training-free speculative decoding that drafts from retrieved n-grams instead of a learned draft model.
miao2023specinfer	system	Generative LLM serving; tree-based speculative verification; distributed and offloading-based inference	End-to-end latency; computational requirement; generative quality preservation	SpecInfer's tree-based speculative inference and parallel verification outperforms existing LLM serving systems by 1.5-2.8x for distributed inference and 2.6-3.5x for offloading-based inference with provably identical generative quality.	Quality of small speculative models bounds acceptance; tree verification adds scheduling complexity	high	full-text	Shows token-tree speculation generalizes speculative decoding to multi-token parallel verification in serving systems.
spector2023accelerating	system	GPT-2 Large (762M params); single-batch on-device inference; tree-structured speculative batches	single-batch decoding latency	Staged speculative decoding (tree-structured batches plus a second speculation stage) reduces single-batch decoding latency by 3.16x on a 762M-parameter GPT-2-L while perfectly preserving output quality.	Demonstrated on a small on-device model; small-batch regime only; gains may not transfer to large-batch datacenter serving.	high	full-text	Evidence that speculative decoding targets the low-arithmetic-intensity, small-batch on-device regime.
chen2023cascade	system	Speculative decoding with neural and statistical drafters; MT-bench-style tasks; multiple target LLMs	speedup over standard speculative decoding; output distribution preservation	Cascade Speculative Drafting (vertical cascade removing autoregressive drafting, horizontal cascade optimizing draft time allocation) adds up to 81% speedup over speculative decoding (per-table speedups around 2.65-2.96x) while preserving the target distribution exactly.	Requires tuning cascade recursion hyperparameters (Knn, lenience); gains vary across tasks and settings.	high	full-text	Pushes speculative drafting beyond autoregressive draft models via cascaded statistical drafting.
sun2024triforce	system	Llama2-7B-128K and Llama2-13B-128K; A100 on-chip; 2x RTX 4090 offloading; batch up to 6; 120K+ contexts	end-to-end speedup, s/token, acceptance rate	TriForce's hierarchical self-speculation (retrieval-based sparse-KV draft plus small-model drafting) achieves up to 2.31x on Llama2-7B-128K on an A100, 7.78x in offloading on two RTX 4090s (0.108s/token, half as slow as the A100 autoregressive baseline), and 4.86x over DeepSpeed-Zero-Inference on one RTX 4090, with a theoretical 13.1x upper bound.	Offloading gains depend on CPU-GPU KV transfer; complex three-cache hierarchy; acceptance-rate-based speedups vary with temperature (though >0.9 at temp 1.0).	high	full-text	Long-context speculative decoding that exploits attention sparsity for drafting, closing the gap between on-chip and offloaded long-sequence serving.
lepikhin2020gshard	system	600B-parameter sparsely-gated MoE multilingual MT Transformer; 100 languages to English; 2048 TPU v3 cores; ~13B training examples	BLEU, training cost (TPU core-years), step-time scaling	GShard scaled a multilingual MT MoE Transformer beyond 600B parameters that trained on 2048 TPU v3 accelerators in 4 days (22 TPU core-years vs 235.5 for the dense 2.3B baseline), with per-step execution time growing only 1.7x when the model scaled 16x.	Training-focused evaluation; quality gains strongest for high-resource languages; TPU/XLA-specific implementation.	high	full-text	Foundational evidence that MoE scales quality sublinearly in compute cost — background for sparse-inference arguments.
fedus2021switch	benchmark	T5-Base/T5-Large scale MoE models up to trillion parameters; C4 corpus; TPU clusters	pre-training speedup (steps/wall-clock) at matched FLOPs	Switch Transformer's simplified top-1 sparse routing yields up to 7x pre-training speedups at the same FLOPs per token, a 4x speedup over T5-XXL at trillion-parameter scale, and a mean 5x step speedup across all 101 multilingual languages (91% of languages at 4x+).	Focus is training efficiency and stability (bfloat16, capacity factor), not serving/inference latency; communication cost grows with expert count.	high	full-text	Canonical sparse MoE scaling result; motivates expert-parallel serving systems and inference-aware MoE design.
hwang2022tutel	system	MoE layers; SwinV2-MoE (vision); 16 to 2,048 A100 GPUs; Fairseq baseline	MoE-layer speedup; end-to-end training/inference speedup	Tutel's adaptive parallelism/pipelining with flexible all-to-all delivers 4.96x and 5.75x single-MoE-layer speedups over 16 and 2,048 A100 GPUs versus prior SOTA, and 1.55x training / 2.11x inference end-to-end on SwinV2-MoE over Fairseq.	Evaluated on a vision MoE model rather than LLM workloads; layer-level speedups may not translate to LLM serving	high	full-text	Evidence that adaptive MoE parallelism matters for scaling expert-parallel inference; relevant to MoE serving background.
rajbhandari2022deepspeed	system	MoE language models; encoder-decoder and autoregressive variants; DeepSpeed library	Model size compression, inference latency and cost vs dense and prior MoE systems	DeepSpeed-MoE's architecture + compression shrinks MoE model size by up to 3.7x, gives 7.3x better latency/cost than existing MoE inference solutions, and serves up to 4.5x faster and 9x cheaper than quality-equivalent dense models.	Pre-dates modern GPU generations and serving stacks; claims are end-to-end system comparisons of their own stack	high	full-text	Early evidence that MoE + expert pruning/quantization yields large latency and cost wins over dense models.
jiang2024mixtral	theoretical	Mixtral 8x7B SMoE, 32k-token context; 8 experts per layer, 2 active per token	active vs total parameters; benchmark accuracy; inference speed characteristics	Mixtral 8x7B gives each token access to 47B parameters while activating only 13B, matching or beating Llama 2 70B and GPT-3.5, with faster inference at low batch sizes and higher throughput at large batch sizes.	Model paper, not a serving study; routing overhead and expert memory footprint (all 47B must be resident) not quantified.	high	full-text	Reference MoE model: sparse activation reduces per-token compute but keeps full parameter memory footprint.
deepseekai2024deepseekv2	system	DeepSeek-V2 (236B total / 21B active), 128K context, 8x H800 node, 8.1T-token pretraining corpus	KV cache size, generation throughput (tokens/s), training cost	Compared with DeepSeek 67B, DeepSeek-V2 saves 42.5% of training costs, reduces KV cache by 93.3%, and boosts maximum generation throughput to 5.76x (over 50K generation tokens/s and 100K prompt tokens/s on a single 8xH800 node), with further 6-bit KV quantization in deployment.	Company tech-report evaluation; comparison is to DeepSeek 67B rather than external baselines; MLA is a proprietary-style architecture.	high	full-text	Production-scale evidence that latent KV compression (MLA) plus MoE sparsity radically cuts cache and raises serving throughput.
deepseekai2024deepseekv3	system	DeepSeek-V3: 671B total / 37B activated parameters MoE; MLA attention; 14.8T pre-training tokens; H800 GPUs	Model architecture efficiency (activated params, KV compression via MLA); training cost	DeepSeek-V3's 671B-parameter MoE activates only 37B per token with Multi-head Latent Attention (which compresses the KV cache) and trains for 2.788M H800 GPU hours, outperforming open models and matching leading closed-source models.	Model report, not a serving-system evaluation; inference latency/throughput numbers not reported in abstract	high	full-text	Production-scale evidence that MLA + MoE architectures cut KV cache and active compute, shaping the inference-optimization design space.
liu2023ring	system	Language modeling and RL tasks; sequences up to millions of tokens across multiple devices	achievable context length; communication-computation overlap	Ring Attention distributes long sequences across devices with blockwise attention/feedforward and fully overlapped KV-block communication, enabling training and inference of sequences up to device-count times longer than prior memory-efficient Transformers without approximation.	Requires multi-device clusters; throughput-per-device not improved, only context scaling; overlap quality depends on bandwidth balance.	high	full-text	Distributed long-context evidence: KV communication can be hidden behind blockwise attention compute.
yang2024context	benchmark	Llama3 405B, 128K-1M context, up to 128 H100 GPUs across 16 nodes (RDMA and TCP)	prefill latency, parallelization efficiency, FLOPs utilization	Ring-attention context parallelism achieves near-linear prefill scaling to 128 H100s: 1M-context prefill of Llama3 405B in 77s (93% parallelization efficiency, 63% FLOPs utilization) and 128K prefill in 3.8s, with similar results over RDMA and TCP.	Requires multi-node GPU fleets; gains are for long-context prefill latency, not decode throughput; single-host 128K baseline already takes ~60s.	high	full-text	Lossless pass-KV/pass-Q ring attention variants make million-token prefill latency practical and are compatible with KV-cache quantization and paging.
shazeer2019fast	benchmark	Transformer 211M/192M, WMT14 En-De, batch 1024, TPUv3, 128-token sequences	incremental decode time per step / per token	Multi-query attention cuts incremental decoder step time from 47ms to 3.9ms per step (amortized 46us to 3.8us per token, about 12x) versus the multi-head baseline with only minor quality degradation.	Authors report small BLEU/perplexity degradation vs baseline; measured on TPUv3 with padded fixed shapes (batch 1024), not production serving stacks.	high	full-text	Origin of MQA, the KV-head-sharing idea that underlies GQA and modern serving-efficient LLMs (memory-bandwidth-bound decode analysis).
ainslie2023gqa	system	Existing multi-head LM checkpoints uptrained to MQA/GQA using 5% of original pre-training compute; grouped-query attention with intermediate number of KV heads	Inference speed vs. output quality trade-off (GQA vs MQA vs multi-head attention)	Uptrained GQA achieves quality close to multi-head attention with speed comparable to MQA, and the uptraining recipe costs only 5% of original pre-training compute.	MQA alone can degrade quality; uptraining still requires some additional compute; abstract reports no absolute latency/token-rate numbers.	high	abstract-only	Establishes GQA as the de-facto KV-cache-reducing attention variant used by nearly all later inference-optimized LLMs.
gu2023mamba	theoretical	Mamba-3B vs same-size Transformers; A100 GPUs; sequences up to million-length	generation throughput; sequence-length scaling	Mamba reports 5x higher generation throughput than same-size Transformers (and up to 3x faster than prior SSM kernels on A100) with linear sequence-length scaling, while Mamba-3B matches the quality of Transformers twice its size.	Architecture paper, not a serving system; throughput measured on its custom kernel, not an end-to-end serving stack; quality parity shown only up to 1B-3B scale.	high	full-text	Evidence that selective SSM architectures remove the KV cache bottleneck entirely, a structural alternative to cache optimization.
song2024powerinfer	system	OPT-30B/OPT-175B and other LLMs; single NVIDIA RTX 4090 consumer GPU	token generation rate; speedup vs llama.cpp	PowerInfer's GPU-CPU hybrid engine (hot neurons on GPU, cold on CPU) outperforms llama.cpp by up to 11.69x on a single RTX 4090, with OPT-30B reaching 82% of an A100's token generation rate.	Abstract-only record; benefits rely on power-law neuron activation locality and may not hold for dense-activation models; CPU-GPU transfer can become bottleneck on other hardware.	high	abstract-only	Demonstrates activation sparsity as a lever for single-GPU/PC-class LLM inference without accuracy loss.
zhou2024survey	survey	LLaMA-2-70B and LLaMA-7B on A100 GPUs; comparative experiments on representative sub-fields	memory footprint, per-token latency, runtime operator share, parameter share	Reports that LLaMA-2-70B in FP16 needs ~140 GB VRAM (~100 ms per generated token on 2 A100s), that attention and linear operators together exceed 75% of inference runtime in their profiling, and that FFN modules account for 63.01% (LLaMA-7B) and 71.69% (LLaMA-70B) of parameters.	Survey's own comparative experiments cover only selected sub-fields; hardware profiling on limited GPU set.	high	full-text	Quantitative anchor points (memory, latency, operator breakdown) for the review's problem-motivation section.
wan2023efficient	survey	n/a — survey spanning model-, data-, and framework-centric efficient-LLM literature (LLaMA-family data points used as illustrations)	taxonomy coverage	Organizes efficient-LLM research into model-, data-, and framework-centric categories and compiles third-party results such as Mixtral 8x7B running ~6x faster than LLaMA-2 70B with 12.9B active parameters per token and KIVI's 2.6x reduction in peak KV-cache memory, without running new experiments itself.	Secondary source; quantitative claims are cited from other papers and not verified here.	high	full-text	Broad taxonomy useful for structuring the review's model-compression vs system sections.
zhu2023survey	survey	LLM compression literature (quantization, pruning, knowledge distillation, low-rank factorization)	taxonomy, metrics (speedup ratio, compression ratio, inference time), benchmarks	Survey of LLM model-compression techniques with no single headline number, but notes a 175B-parameter model such as GPT-3 needs at least ~350GB in FP16 and roughly five 80GB A100s for inference.	Descriptive survey; no original experiments; numbers cited from primary sources.	high	full-text	Useful map of quantization/pruning/KD landscape and evaluation metrics for the compression side of inference optimization.
chittyvenkata2023survey	survey	Transformer families (BERT, GPT, ViT); algorithmic (distillation, pruning, quantization, NAS) and hardware-level techniques	Qualitative coverage of inference-optimization techniques; parameter/FLOPs-accuracy tradeoffs	Comprehensive survey of transformer inference optimization across algorithmic and hardware levels; summarizes quantitative parameter/FLOPs/accuracy tradeoffs without reporting a single headline performance number.	No quantitative speedup/latency claims in the abstract; survey dated relative to LLM-era serving systems	moderate	abstract-only	Broad background taxonomy for the review's algorithmic-level (quantization/pruning/distillation) and accelerator sections.
liu2025lmcache	system	vLLM and SGLang engines; multi-round QA and document analysis workloads; GPU/CPU/storage/network cache layers	throughput, latency, prefix cache hit ratio, KV transfer bandwidth	Combining LMCACHE with vLLM yields up to 15x throughput improvement (and at least 2x lower latency across local prefix caching, distributed reuse, and PD disaggregation), while industry context truncation can cut prefix cache hit ratio by half.	Authors report transfer throughput up to ~49GBps only with batched/pipelined ops vs sub-1GB/s naive serialization; adoption insights are enterprise-observational.	high	full-text	Open-source KV cache offloading/sharing layer enabling cross-engine prefix reuse and PD disaggregation; key infrastructure for cache-centric serving.
cai2024pyramidkv	benchmark	LLaMA-3-8B-Instruct, LLaMA-3-70B-Instruct, Mistral-7B on LongBench (17 datasets, avg 1,235-18,409 tokens); KV cache sizes 64-256	LongBench accuracy, Needle-in-a-Haystack Acc, KV memory	PyramidKV matches full-KV-cache performance while retaining only 12% of the KV cache on LongBench, beats other compression methods at extreme 0.7% cache (up to +20.5 absolute accuracy on TREC), and reaches 100.0 Acc on Needle-in-a-Haystack with just 128 KV entries for LLaMA-3-70B.	Pyramidal allocation is a heuristic derived from observed attention patterns; slightly worse than baselines on some saturated tasks at the smallest cache sizes.	high	full-text	Layer-wise KV budget allocation as a memory-saving strategy distinct from uniform eviction.
fu2024break	system	LLMs on MT-Bench and code-completion tasks; single and multiple GPUs; no auxiliary draft model	Decoding speedup; parallelism scaling	Lookahead decoding, an exact parallel decoding algorithm without draft models or data stores, speeds up autoregressive decoding by up to 1.8x on MT-bench and up to 4x with strong scaling on multiple GPUs for code completion.	Speedup sensitive to n-gram predictability of the text; extra FLOPs per step traded for fewer steps	high	full-text	Draft-model-free alternative to speculative decoding, relevant for deployments where drafts are unavailable.
lin2024infinite	system	Context lengths from a few to 2,000K tokens; 32 A100 GPUs cluster	System throughput; memory utilization; supported context length	Infinite-LLM disaggregates attention layers and pools GPU memory across the cluster, improving throughput by 1.35-3.4x versus state-of-the-art methods on contexts up to 2,000K tokens.	Cluster-scale complexity; attention-layer disaggregation adds scheduling and transfer overheads	high	full-text	Shows layer-level disaggregation plus pooled memory as a route to elastic long-context serving.
cheng2024recurrent	system	Vicuna models on MT-Bench; PyTorch on NVIDIA H100; MLX on Apple M2 Ultra Metal GPU	end-to-end speedup, tokens/step	ReDrafter, using an RNN draft model conditioned on LLM hidden states with dynamic tree attention over beam search and knowledge distillation, accelerates Vicuna inference by up to 2.8x on H100 (PyTorch) and up to 2.3x on Apple Silicon Metal GPUs (MLX).	Requires training the drafter via distillation; beam width/length must be tuned per hardware; recurrence reduces draft-stage GPU parallelism.	high	full-text	State-of-the-art speculative decoding via recurrent drafting; strong H100 and on-device numbers.
wei2023outlier	system	BERT, OPT, BLOOM, BLOOMZ, and LLaMA under standard and fine-grained quantization at 8-bit, 6-bit, and 4-bit	Quantization accuracy (near-floating-point performance; task metrics)	Outlier Suppression+ achieves near-floating-point performance at 8-bit and 6-bit standard quantization and sets a new state of the art for 4-bit BERT with a 15.5% improvement.	Focuses on activation outliers in PTQ; gains vary by model family and bit-width; abstract reports no inference speedup numbers.	high	abstract-only	Equivalent shifting/scaling tames asymmetric activation outliers, a key enabler for low-bit PTQ of LLMs.
li2025eagle	system	Chat and reasoning models evaluated on five tasks; SGLang framework at batch size 64	Speculative-decoding speedup ratio and serving throughput	EAGLE-3 achieves a speedup ratio up to 6.5x (about 1.4x over EAGLE-2) and a 1.38x throughput improvement in SGLang at batch size 64.	Draft model must be trained (training-time test); gains depend on acceptance rates and batch; abstract reports peak ratios, not average across all tasks.	high	abstract-only	Scales speculative drafting via direct token prediction and multi-layer feature fusion, showing speedups improve with more training data.
agrawal2024vidur	simulation	LLaMA2-70B and other LLMs; workload/config search space (parallelization, batching, scheduling); CPU machine for search	Predicted vs measured latency/throughput (fidelity error); configuration-search cost	Vidur simulates LLM inference with less than 9% latency error across tested LLMs, and Vidur-Search finds the best LLaMA2-70B deployment configuration in 1 hour on a CPU machine versus an estimated 42K GPU-hours (~$218K) of deployment-based exploration.	Simulation fidelity depends on profiling accuracy; validated range of models/hardware is limited	high	full-text	Enables cheap configuration search (batching/scheduling/parallelism) without expensive GPU experiments; key methodology for the review.
wang2025burstgpt	case study	10.31 million request traces from regional Azure OpenAI GPT services over 213 days	Workload characteristics: request concurrency burstiness, conversation patterns, response lengths, and system failures	BurstGPT documents 10.31M real production traces over 213 days showing frequent burstiness variations that expose declines in efficiency, stability, and reliability of realistic LLM serving.	Observational dataset, not an optimization; traces reflect one provider's service mix; no speedup or latency metrics are claimed.	high	abstract-only	Provides the realistic workload grounding needed to evaluate KV cache, scheduling, and disaggregation optimizations beyond synthetic assumptions.
yuan2024llm	survey	LLM inference literature; roofline analysis of Llama-13B (~26GB FP16 weights) and other models; LLM-Viewer tool	Roofline-based bottleneck analysis (memory-bound vs compute-bound) of inference techniques	Survey introduces a roofline-model framework showing why LLM inference is memory-bound and analyzes compression, early-exit, MoE, and system-level techniques, e.g., noting Llama-13B occupies ~26GB in FP16 and that contextual sparsity can reach 80%.	Survey scope; no new empirical results; qualitative treatment of system-level serving	high	full-text	Provides the roofline framing that organizes the review's discussion of memory-bound decode vs compute-bound prefill.
li2024llm	survey	n/a — survey of LLM serving systems since 2023 (Andes, FlashDecoding++, Parrot, FlashAttention-3, etc.)	coverage taxonomy of system-level serving optimizations	Reviews post-2023 system-level LLM serving optimizations that improve performance without altering decoding mechanisms (e.g., QoE-based token scheduling in Andes, flat-GEMM double buffering in FlashDecoding++, warp-specialized FlashAttention-3), reporting qualitative gains rather than new experiments.	Scope limited to system-level changes, excluding decoding-mechanism and model-level work; third-party numbers not reproduced in this text.	high	full-text	Map of the serving-systems landscape (batching, disaggregation, scheduling) for the review's system-level section.
hu2024inference	system	LLM serving workloads mixing prefill/decode of different lengths (TetriInfer)	TTFT, JCT, resource usage, performance per dollar	TetriInfer uses 38% fewer resources while lowering average TTFT by 97% and average JCT by 47%, and its measurements show naive mixing can cause 10x prefill slowdowns, 5x prefill-decode slowdowns, and 16% decode throughput loss.	Authors note chunk padding and predicted-resource scheduling add complexity; headline numbers are for their evaluated workload mixes.	high	full-text	Quantifies prefill-decode interference and validates chunked prefill + prefill/decode disaggregation plus two-level scheduling.
forys2026when	simulation	Agentic multi-turn tool-calling workloads; current GPUs and hypothetical stage-specialized NPUs (Vera-Rubin, Groq LPU style)	simulated serving throughput; latency	HeteroPanacea simulations show prefill-decode disaggregation raises serving throughput by up to 75% over traditional serving on current GPUs, and stage-specialized hardware yields up to 2.06x throughput on agentic workloads.	Simulation results, not measured on real hardware; 4-way PDAF (prefill/decode x attention/FFN) gains assume custom NPUs; no public release at time of writing.	high	full-text	Forward-looking simulation evidence that heterogeneous/disaggregated hardware, not just software, is needed for agentic inference.
wang2025prefill	system	LLM serving workloads under varied TTFT/TPOT SLOs; differentiated-capability GPU instances (prefill-heavy vs decode-heavy)	Goodput (SLO-satisfied throughput); TTFT; TPOT	TaiChi, unifying PD aggregation and disaggregation with latency shifting, improves goodput by up to 77% over state-of-the-art systems under balanced TTFT and TPOT SLOs.	Requires two GPU instance classes and tuning of three sliders; comparison scope limited to tested SLO regimes	high	full-text	Resolves the aggregation-vs-disaggregation debate: optimal regime depends on SLO mix, and hybrid modes win under balanced SLOs.
gao2025duetserve	system	Qwen3-8B and Qwen3-14B on H100 GPUs; workloads of 8000 input / 200 output tokens; token budgets ~2048 (A100) / 8192 (H100)	total throughput, TBT, TTFT	DuetServe improves total throughput by up to 1.3x over state-of-the-art frameworks while maintaining low generation latency, by dynamically activating SM-level prefill/decode spatial partitioning within a single GPU only when predicted TBT degradation threatens SLOs.	Evaluated on only two Qwen models; single-GPU scope; benefits bounded by TPC/SM partitioning granularity (20% of SMs already reach ~60% of peak HBM bandwidth).	high	full-text	Adaptive middle ground between phase aggregation and disaggregation for the serving-systems chapter.
li2026tetris	system	disaggregated long-context serving cluster; varying-length online requests (CDSP chunkwise sequence parallelism)	TTFT, median time-between-tokens (TBT), max request capacity	Tetris, built on Chunkwise Dynamic Sequence Parallelism (CDSP), achieves up to 4.35x lower TTFT under max sustainable loads, reduces median TBT by up to 40.1%, and increases max request capacity by up to 45% versus state-of-the-art systems.	Abstract-only access locally (no full text); numbers are headline results from the abstract, deployment details unverified.	moderate	abstract-only	Fine-grained intra-request sequence-parallelism scheduling that exploits fragmented resources in disaggregated serving.
lai2025tokenscale	system	Llama and Qwen models of various sizes, tensor-parallel degrees; NVIDIA A100 cluster; Azure production traces (bursts in 47% of operational time, avg 2.3s)	TTFT/TPOT SLO attainment; serving cost	TokenScale (Token Velocity predictive metric + Convertible Decoders) improves SLO attainment from 50-88% to 80-96% and cuts costs by 4-14% versus DistServe, BlitzScale, and AIBrix under production traces.	Prototype built on vLLM/LMCache; evaluation on traces, not full production; convertible decoders trade decode capacity during bursts.	high	full-text	Evidence that disaggregated serving needs proactive, token-level autoscaling rather than lagging utilization metrics.
li2026not	system	4x NVIDIA H100 80GB NVLink node; multi-turn chat/agent workloads; batch sizes up to 200; contexts up to 32K-64K tokens	Turn 2+ TTFT, TPOT, throughput (tokens/sec)	PPD (Prefill-capable Decode) dynamic routing reduces Turn 2+ TTFT by ~68% on average (up to 73.3% in 1P_3D configs) while keeping TPOT competitive, exploiting that append-prefill slows co-located decode by only ~2% vs ~48% for full prefill at batch size 200.	Single-node experiments with bandwidth simulation for inter-node effects; routing weights must be tuned per SLO; no single static policy satisfies all SLOs.	high	full-text	Quantifies append-prefill vs full-prefill interference, informing PD-disaggregation design choices.
she2026laps	system	Qwen2.5-32B on H200 GPUs; multi-turn workloads with long (>1K token) vs short (<64 token) prefills	prefill latency (TTFT), SLO violation rate, request throughput	LAPS reduces prefill latency by over 30% vs vanilla SGLang under prefill-decode disaggregation, cuts SLO violations by 28% (data-parallel) and a further 12% vs the SGLang load-balancing router, and improves request throughput by 35% at high concurrency.	Numbers are workload- and model-specific (Qwen2.5-32B, multi-turn traces); 2026 preprint; gains depend on prompt-length heterogeneity.	high	full-text	Extends PD disaggregation with length-aware prefill scheduling (long/short prefill separation, dual-queue, CUDA-Graph batching).
ye2025flashinfer	system	KV-cache block-sparse/composable formats; JIT-compiled attention templates; integrated in SGLang, vLLM, MLC-Engine; long-context and parallel-generation workloads	Inter-token latency, long-context latency, parallel-generation speedup at kernel and end-to-end levels	FlashInfer's customizable attention engine reduces inter-token latency by 29-69% versus compiler backends, cuts long-context inference latency by 28-30%, and gives 13-17% speedups for parallel generation.	Kernel-level gains depend on workload shape and integration; comparison baselines vary by benchmark	high	full-text	Evidence that attention-kernel engineering (block-sparse KV formats, JIT) delivers large decode-latency wins across serving stacks.
lee2024infinigen	system	OPT-13B, Llama-2-13B; RTX A6000 48GB + CPU DDR4-2666 96GB, PCIe 3.0; PG-19, WikiText-2, PTB; KV budgets 10-20%	end-to-end speedup, perplexity	InfiniGen improves overall performance of offloading-based long-text inference by up to 3.00x over prior KV cache management methods while offering substantially better model accuracy, by speculatively prefetching only the essential KV entries needed for the next attention layer.	Targets offloading-based serving only; speculation overhead and accuracy of token-importance prediction limit gains; tested on 13B-class models.	high	full-text	Selective KV prefetch as a bridge between offloading and cache management for long contexts.
strati2024d	system	Large models (GPT-3/OPT/BLOOM class) in pipeline-parallel cloud deployments	pipeline bubble reduction; GPU memory utilization; recovery time	DejaVuLib-based prompt-token disaggregation, microbatch KV swapping, and state replication address pipeline bubbles (prompt vs token latency differs by up to 2 orders of magnitude), GPU memory overprovisioning, and long failure recovery.	Abstract and intro are qualitative; no headline end-to-end speedup numbers reported in the skimmed text; effects depend on pipeline configuration.	moderate	full-text	KV cache streaming as a unifying primitive for disaggregation, memory management, and fault tolerance in distributed serving.
liu2024droidspeak	system	Llama-3-8B/70B, Mistrallite, MAmmoTH2 model pairs; A100 GPUs; 5K-40K token inputs	throughput, prefill latency (TTFT), quality (F1, Rouge-L, code similarity)	DroidSpeak achieves up to 4x throughput improvement and about 3.1x faster prefill with negligible quality loss by reusing KV caches across different same-architecture LLMs, e.g., cutting a 40K-token prefill on A100 from 4s to 0.08s.	Requires same-architecture model pairs; selective layer recomputation needed for quality; quality measured on a limited benchmark set.	high	full-text	First cross-model KV cache reuse system (selective layer recomputation + pipelined loading) for compound/agentic multi-LLM serving.
chen2024kvdirect	system	7B and 70B models; multi-GPU nodes (8-GPU node = 160GB memory per worker example); distributed disaggregated prefill/decode; diverse workloads	Per-request latency; KV-cache transfer efficiency; TTFT/TBT	KVDirect's tensor-centric, pull-based KV-cache transfer for distributed disaggregated inference reduces per-request latency by 55% versus baseline under the same resource constraints, and shows naive message-passing achieves only 13.6% effective communication.	Requires custom communication library; gains depend on inter-node bandwidth and workload mix	high	full-text	Key evidence that prefill-decode disaggregation can be extended beyond a single node, addressing the scalability limit of DistServe-style designs.
pengju2026spectrumkv	system	Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Gemma-2-9B-it; WikiText-2 perplexity and NIAH retrieval at 4096-token context; PD-disaggregated transfer	perplexity at fixed KV transfer budget; NIAH retrieval accuracy; TTFT	SpectrumKV's per-token FP16/INT8/INT4 transfer changes WikiText-2 perplexity by only +1.97%/-0.06%/-0.44% at a 50% KV budget versus +25.85%/+22.07%/+35.63% for PDTrim, reaches 52.6% vs 26.3% NIAH retrieval at b=0.3 (100% by b=0.5), and cuts transfer-path TTFT by 50-62% at b=0.5.	INT4 tolerance is model-dependent (Qwen2.5-7B fails INT4 KV; probe required); results specific to the tested budget levels and PD setting.	high	full-text	Argues PD KV transfer should be treated as a precision-allocation problem rather than binary token selection.
qiu2024efficient	system	Real-world datasets and production workload traces; models with 1.3-11 GB footprints (single-GPU serving)	average job completion time, throughput	Speculative shortest-job-first (SSJF) scheduling with a light proxy model for output-length prediction reduces average JCT by 30.5-39.6% and increases throughput by 2.2-3.6x versus FCFS schedulers across no batching, dynamic batching, and continuous batching settings.	Depends on proxy-model length prediction accuracy; evaluated on single-GPU serving stacks.	high	full-text	Length-aware scheduling addresses head-of-line blocking from autoregressive length variability.
zheng2023response	system	Vicuna (LLaMA-based) on real-world instruction datasets	inference throughput	Response-length-aware sequence scheduling that groups queries with similar predicted response lengths into micro-batches improves inference throughput by 86% over vanilla batch inference without compromising quality, where vanilla batching wastes 66% of generated tokens on redundant completion.	Relies on LLM length-perception accuracy (GPT-4/Claude-level predictors achieve Acc-100 >90%); evaluated on a single model family.	high	full-text	Early demonstration that output-length prediction enables smarter batching — precursor to SJF schedulers.
xiao2024duoattention	system	MHA (Llama) and GQA models; long-context inference; synthetic-data head identification; combined with quantization on A100	Inference memory, decoding/prefill latency, accuracy, max context length	DuoAttention applies full KV cache only to retrieval heads and constant-length cache to streaming heads, reducing long-context memory by up to 2.55x (MHA) / 1.67x (GQA), speeding decoding up to 2.18x/1.50x and prefill up to 1.73x/1.63x, and with quantization runs Llama-3-8B at 3.3M context on one A100.	Requires head-identification optimization pass on synthetic data; assumes retrieval-head sparsity holds for the target tasks	high	full-text	Strong evidence that head-level specialization (retrieval vs streaming) makes long-context serving dramatically cheaper.
gu2024when	theoretical	LMs up to 1B parameters; pretraining data from 50M to 5B tokens; softmax vs sigmoid attention	attention-sink metric, validation loss	Attention sinks emerge only after effective optimization on sufficient training data and act like key biases storing extra attention scores, and replacing softmax normalization with sigmoid attention eliminates the sink (metric drops from 45.11% to 2.46% at 1B params) at negligible loss cost (3.10 vs 3.07).	Analysis limited to models up to 1B parameters; emergence conditions demonstrated correlationally across data size, loss, and architecture variants.	high	full-text	Theoretical grounding for attention-sink-based optimizations (StreamingLLM-style KV tricks, quantization).
ribar2023sparq	system	Llama 2/3, Mistral, Gemma, Pythia; wide range of downstream tasks; long-sequence batched inference	Attention data-transfer volume; throughput; accuracy	SparQ Attention's selective fetching of the cached history cuts attention data transfers by up to 8x without substantial accuracy drops, on off-the-shelf LLMs with no fine-tuning.	Approximate attention may hurt tasks needing exact full-context recall; speedup depends on attention sparsity of the workload	high	full-text	Demonstrates bandwidth-side KV optimization: predicting high-attention tokens can yield large transfer savings in decode.
ge2023model	system	Llama 1 models; math/code/knowledge/reasoning benchmarks; 8x A100 80GB	KV-cache memory reduction; attention-score recovery; generation quality	FastGen's adaptive head-wise KV cache compression (evicting local-context, special-token, and broad-attention patterns per head) compresses 35% of the cache while recovering over 95% of attention scores, with negligible quality loss.	Requires lightweight attention profiling per model; compression ratio modest vs later eviction schemes	high	full-text	Early evidence that head-specific KV eviction policies can cut memory with negligible quality loss, precursor to retrieval-head methods.
chen2024nacl	system	Long-context LLMs with KV cache eviction during the encoding phase; combines proxy-token (attention-statistics) and random eviction	Task performance on short/long-text tasks and KV cache memory reduction	NACL improves short- and long-text task performance by 80% and 76% respectively while reducing KV cache by up to 50% with over 95% performance maintenance.	Authors criticize prior eviction works for biased local attention statistics and inadequate short-text perplexity-only evaluation; abstract-only, so eviction robustness at extreme compression unverified here.	high	abstract-only	Single-pass encoding-phase KV eviction framework showing cache can be halved with near-full task performance.
eliseev2023fast	system	Mixtral-8x7B and Mixtral-8x7B-Instruct; T4, RTX 3060, RTX 3080 Mobile (11-16GB VRAM); PCIe Gen3 8-16GB/s	tokens per second, inference latency	Enables interactive Mixtral-8x7B inference at 2-3 tokens per second on consumer GPUs by offloading experts with an LRU cache (k=2 experts cached on 12GB GPUs, k=4 on 16GB) plus mixed HQQ quantization, with speculative expert prefetching to hide loading latency.	Achieves interactive but not fast speeds; sub-1-bit QMoE compression caused too large perplexity loss for Mixtral; LRU cache ignores expert-activation patterns.	high	full-text	Shows MoE expert offloading + LRU caching makes large MoE models usable on commodity hardware.
shen2025efficient	system	Llama2, Llama, GPT-NeoX, Falcon-7B (6B-20B); 4th-gen Intel Xeon CPUs; INT4 weight-only quantization (GPTQ/AWQ/SignRound/TEQ recipes)	Per-token generation latency on CPUs; accuracy vs FP32	Automatic INT4 weight-only quantization plus an optimized CPU runtime achieves 20-80 ms per-token generation latency for 6B-20B LLMs with accuracy within ~1% of FP32.	CPU-only target; latency figures are for next-token generation under a proxy config (input=output=32 tokens), not end-to-end serving	high	full-text	Evidence that weight-only INT4 plus AMX kernels makes LLM inference viable on commodity CPUs, relevant to the hardware-aware quantization thread.
xu2023llmcad	system	on-device mobile GPUs; small memory-resident LLM + larger verifier LLM (e.g., LLaMA variants)	token generation speed vs existing on-device engines	LLMCad achieves token generation speeds up to 9.3x faster than existing inference engines on mobile devices, where weight reloading alone (the memory wall) can lengthen latency by 59-224x.	Requires a compatible memory-resident small model; gains depend on small-model accuracy and tree-verification acceptance; mobile-hardware-specific.	high	full-text	On-device speculative decoding via token trees with self-adjusting fallback and compute-IO pipelining; shows speculative methods can break the mobile memory wall.
alizadeh2024llm	system	LLMs larger than available DRAM, parameters stored on flash memory; CPU and GPU inference; sparsity-aware activation reuse	Inference speed vs naive flash-to-DRAM loading; model size supportable per DRAM	Windowing and row-column bundling let models up to twice the size of available DRAM run with 4-5x (CPU) and 20-25x (GPU) inference speed increases versus naive loading.	Assumes activation sparsity and flash-friendly access patterns; speeds are relative to naive loading, not to full-DRAM baselines; abstract-only.	high	abstract-only	Hardware-informed flash/DRAM orchestration that makes very large models feasible on memory-limited edge devices.
yang2024pyramidinfer	system	LLM inference with layer-wise KV retention; baseline Accelerate; chat-style real-time workloads	Throughput and GPU memory reduction in KV cache	PyramidInfer improves throughput by 2.2x compared to Accelerate while reducing KV cache GPU memory by over 54% without sacrificing performance.	Exploits inter-layer dependency in attention weights, which may not transfer to all architectures; comparison is against Accelerate baseline only.	high	abstract-only	Layer-wise crucial-context retention compresses KV cache during pre-computation, cutting both memory and compute.
wu2024layer	system	Large transformer LMs where only a small number of layers compute and cache KVs	Inference throughput and memory consumption vs standard transformers	Layer-condensed KV caching (computing KVs for only a few layers) achieves up to 26x higher throughput than standard transformers with competitive language modeling and downstream performance.	Quality depends on which layers retain KVs and on model depth; abstract-only, so degradation at scale unquantified here.	high	abstract-only	Orthogonal to other memory-saving techniques; drastic throughput gains from caching KVs at a subset of layers.
ye2024chunkattenti	system	Multi-tenant LLM serving with shared system prompts of length 1024-4096; KV tensors chunked into a prefix tree	Self-attention kernel speedup vs state-of-the-art implementation	ChunkAttention speeds up the self-attention kernel by 3.2-4.8x versus the state-of-the-art implementation for system-prompt lengths of 1024 to 4096.	Speedup is kernel-level and depends on prefix sharing across requests; abstract reports no end-to-end serving latency/throughput.	high	abstract-only	Prefix-aware KV cache sharing converts redundant system-prompt compute into memory and kernel-level wins.
yuan2024kv	benchmark	10+ state-of-the-art long-context efficiency approaches across seven categories (quantization, token dropping, prompt compression, linear-time models, hybrid architectures)	Comparative quality/efficiency metrics across long-context task categories	Provides a taxonomy and aligned-environment evaluation of 10+ KV-cache compression and long-context approaches across seven task categories, revealing previously unknown phenomena and trade-offs.	Abstract gives no headline numbers; findings are descriptive and depend on benchmark task selection and environment alignment choices.	moderate	abstract-only	First reasonably aligned head-to-head benchmark of long-context efficiency methods, useful for situating KV compression claims.
gong2024llmc	benchmark	Dozens of quantization algorithms, models, and hardware; integer and floating-point, fixed-bit and mixed-precision, LLM and vision-language models	Fair cross-method comparison of quantization accuracy/efficiency trade-offs	LLMC benchmark systematically explores quantization across calibration data, three algorithm strategies, and data formats, providing insights and practical guidance, though the abstract reports no aggregate numeric result.	Abstract contains no quantitative findings; benchmark coverage and hardware support bound generality.	moderate	abstract-only	Standardized plug-and-play toolkit enabling fair comparison of LLM quantization methods in a fragmented literature.
xia2024unlocking	survey	Speculative decoding literature: drafting and verification strategies across leading methods	Qualitative taxonomy and comparative analysis (no single metric)	Provides a formal definition and formulation of speculative decoding, in-depth analysis of drafter selection and verification strategies, and a comparative analysis of leading methods under third-party testing environments.	Survey abstract reports no quantitative speedup numbers; coverage depends on methods selected and third-party test setups.	moderate	abstract-only	Reference survey structuring the speculative decoding landscape for the review's decoding-acceleration chapter.
elhoushi2024layerskip	system	Llama models of several sizes under pretraining, continual pretraining, and finetuning regimes; CNN/DM, coding, and TOPv2 tasks	Inference speedup from early-exit + self-speculative decoding	LayerSkip's self-speculative decoding achieves speedups of up to 2.16x on CNN/DM summarization, 1.82x on coding, and 2.0x on TOPv2 semantic parsing.	Requires training-side changes (layer dropout + early-exit loss); gains task-dependent; no auxiliary modules but speedups modest versus draft-model methods.	high	abstract-only	Training recipe plus self-speculative decoding that reuses the same model as drafter and verifier with shared activations.
svirschevski2024specexec	system	50B+ parameter LLMs (e.g., Llama-2-70B) on consumer GPUs (RTX-class) with RAM offloading; 4-bit and 16-bit weights; MTBench	Tokens per second; tokens generated per target-model iteration; speedup vs sequential offloaded inference	SpecExec generates up to 20 tokens per target-model iteration, running 50B+ models on consumer GPUs with RAM offloading at 4-6 tokens/s (4-bit) or 2-3 tokens/s (16-bit), a 10-18x speedup over sequential offloaded inference.	Relies on token-probability spikiness and draft-target alignment; offloading setting only, not datacenter serving	high	full-text	Demonstrates speculative execution's largest gains in bandwidth-bound offloaded settings, extending speculation to consumer hardware.
li2024nearest	system	Llama-2-Chat 7B/13B/70B; WikiText-103, NQ, Biography, MMLU, Pile-of-Law, TruthfulQA; two-stage retrieval (40 passages, 1024 tokens)	inference speedup, ROUGE-1, FActScore, attribution	NEST achieves a 1.8x inference-time speedup on Llama-2-Chat 70B while improving quality and attribution over the base LM (+42.3% ROUGE-1 on WikiText-103 and +21.6% FActScore on Biography) via token-level retrieval, confidence-based interpolation, and relaxed speculative decoding of retrieved spans.	Requires a key-value datastore and on-the-fly passage encoding (latency grows linearly with passage count); no statistical significance tests reported.	high	full-text	Speculative decoding meets semi-parametric retrieval — speed plus attribution for knowledge-intensive generation.
zhong2024propd	system	multiple datasets, LLMs, and batch sizes (ProPD token-tree pruning; e.g., Medusa-style verification)	decoding speedup vs existing parallel-decoding algorithms	ProPD's early tree-pruning and dynamic token-tree generation consistently outperform existing parallel-decoding algorithms (e.g., Medusa) by 1.1-3.2x across datasets, models, and batch sizes.	Abstract-only access locally; pruning overhead and worst-case behavior not quantified in the available text.	moderate	abstract-only	Improves tree-based parallel decoding by pruning unpromising branches and adapting tree shape to batch size and task.
park2024improving	system	Multiple decoder-based LLMs incl. OPT-30B; commodity CPU+GPU hardware; dynamic workload allocation	Throughput vs state of the art	A finely tuned dynamic CPU-GPU workload allocation technique for decoder models on commodity hardware increases throughput by up to 105% (OPT-30B) versus the state of the art.	Abstract-only record; mechanism details and reproducibility not verifiable locally	high	abstract-only	Evidence that harvesting idle CPU compute alongside GPU offloading can nearly double commodity-hardware throughput.
dai2024deepseekmoe	system	DeepSeekMoE at 2B, 16B, and 145B parameters; fine-grained expert segmentation (mN experts, mK activated) plus Ks shared experts	Perplexity/performance per unit computation vs dense and GShard baselines	DeepSeekMoE 16B matches LLaMA2 7B performance with only about 40% of the computation, and the 145B model matches DeepSeek 67B using 28.5% (possibly 18.2%) of the computation.	Specialization benefits shown mainly at training-time quality-per-compute; abstract gives no serving latency/throughput numbers for the MoE routing overhead.	high	abstract-only	Architectural template (fine-grained + shared experts) later deployed in DeepSeek-V2/V3 serving systems.
yao2024exploiting	system	GPT MoE models with 8-64 experts; distributed GPU systems with varying topologies	cross-GPU routing latency; inference throughput	ExFlow's context-coherent expert placement (single Alltoall instead of two) reduces cross-GPU routing latency for up to 67% of tokens and improves inference throughput up to 2.2x over DeepSpeed-MoE without fine-tuning or accuracy loss.	Abstract-only record; relies on inter-layer expert affinity that may not hold for all MoE training regimes; integer-programming placement adds offline planning cost.	high	abstract-only	Shows expert placement exploiting inter-layer affinity can halve MoE all-to-all communication overhead.
ma2024hpipe	system	LLaMA-7B and GPT3-2B on heterogeneous commodity devices with unreliable interconnects; sequence-token-dimension pipelining	End-to-end latency and throughput speedup	HPipe achieves an impressive speedup in latency and throughput of up to 2.28x for long-context inference on heterogeneous commodity devices.	Targets constrained micro-enterprise/individual deployments; heterogeneity and expensive communication cap gains; abstract-only, no absolute token rates.	high	abstract-only	Pipeline parallelism adapted to heterogeneous low-cost hardware for long-context private deployment.
fang2024usp	system	LLAMA3-8B (208K sequence length) and LLAMA2-7B; two 8xA800 nodes (400GB/s NVLink, 1.6Tbps RDMA); also 8xL20 and 8xA100-SXM4	MFU, FLOPS per GPU, iterations/sec	Unified sequence parallelism (USP) achieves 47% MFU for LLAMA3-8B training at 208K sequence length on two 8xA800 nodes and outperforms SP-Ring by 13% and 12% in throughput at 64K and 80K sequence lengths.	Evaluation is training-oriented; MFU reported for a single configuration; optimal Ulysses/ring degree split depends on interconnect topology (e.g., degree-8 Ulysses best on A100 NVLink).	high	full-text	Shows sequence-parallelism design space (Ulysses vs Ring vs hybrid) for long-context compute.
du2021glam	theoretical	GLaM 1.2T-parameter MoE (64 experts/layer) vs GPT-3 175B; 29 NLP tasks	inference FLOPs per token; training energy; zero/one/few-shot accuracy	GLaM's sparsely activated MoE (1.2T params, ~7x GPT-3's size) consumes 1/3 of GPT-3's training energy and about half the inference FLOPs per token while achieving better average zero/one/few-shot performance across 29 tasks.	Training-focused paper; reports FLOPs/energy, not serving latency or throughput; expert routing overheads not modeled.	high	full-text	Early scaling evidence that sparse MoE cuts per-token inference compute at constant quality.
su2023synergy	benchmark	Multiple LLMs and GPU architectures; batched speculative decoding across batch sizes and speculation lengths	speculative decoding latency/throughput vs speculation length and batch size	Characterization shows optimal speculation length shrinks as batch size grows, and the proposed adaptive strategy matches or beats fixed-length speculative decoding, adding 9% latency reduction on time-varying request streams.	Prototype-level implementation; mapping from batch size to optimal length is built by pre-deployment profiling; gains modest.	high	full-text	Evidence that batching and speculative decoding interact, so speculation length must be batch-adaptive.
zhang2024draft	system	LLaMA-2 and variants; layer-skipping for drafting, full-model verification in one forward pass	End-to-end speedup with exact output preservation	Draft & Verify self-speculative decoding achieves speedup up to 1.99x on LLaMA-2 and variants while guaranteeing outputs identical to the unaltered LLM.	Speedup ceiling limited by layer-skip drafting quality and acceptance rate; gains reported as up-to rather than average.	high	abstract-only	Lossless, training-free, zero-extra-memory self-speculative decoding via layer skipping.
kim2023speculative	system	NVIDIA T4 GPU; IWSLT 2017 De-En, WMT 2014 De-En translation, XSUM and CNN/DailyMail summarization	latency speedup, BLEU/ROUGE quality	Big Little Decoder (BiLD) achieves up to 2.12x speedup with ~1-point quality degradation (1.85x with no degradation) by running a small model autoregressively and invoking the large model non-autoregressively only for fallback and rollback corrections.	Speedup ceiling set by small-model accuracy and fallback policy; quality trade-off at maximum speedup; evaluated on T4-class hardware.	high	full-text	Big-little speculative decoding framework — early evidence that draft-then-verify accelerates decoding.

Swipe sideways to see all columns.

References

Kwon, Woosuk et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention — Proceedings of the 29th Symposium on Operating Systems Principles. Abstract only. PagedAttention/vLLM is the reference point for KV-cache memory management and continuous batching in nearly all later serving work.doi:10.1145/3600006.3613165
Agrawal, Amey et al. (2023). SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — arXiv (Cornell University). Full text read. Origin of chunked-prefill + decode-maximal batching now standard in vLLM-class serving engines.doi:10.48550/arxiv.2308.16369
Agrawal, Amey et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — arXiv (Cornell University). Full text read. Introduces chunked prefill + stall-free batching, the scheduling recipe adopted by vLLM and most modern serving engines.doi:10.48550/arxiv.2403.02310
Zhong, Yinmin et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — arXiv (Cornell University). Full text read. Foundational PD-disaggregation system; defines per-phase SLO (TTFT/TPOT) as the serving objective.doi:10.48550/arxiv.2401.09670
Patel, Pratyush et al. (2023). Splitwise: Efficient generative LLM inference using phase splitting — arXiv (Cornell University). Full text read. Foundational evidence for phase-disaggregation: decode does not need latest-GPU compute, enabling cheaper, power-efficient clusters.doi:10.48550/arxiv.2311.18677
Wu, Bingyang et al. (2024). LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism — Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. Abstract only. Argues static parallelism strategies waste resources under variable-length requests — dynamic parallelism for serving.doi:10.1145/3694715.3695948
Duan, Jiangfei et al. (2024). MuxServe: Flexible Multiplexing for Efficient Multiple LLM Serving. — CoRR. Full text read. Multi-model serving via phase-aware colocation — relevant to heterogeneous endpoint fleets.doi:10.48550/arxiv.2404.02015
Zheng, Lianmin et al. (2024). SGLang: Efficient Execution of Structured Language Model Programs — SOSP 2024. Full text read. Canonical system evidence that structured KV cache reuse (radix tree, LRU eviction) accelerates multi-call LLM programs.doi:10.1145/3694715.3695950
Qin, Ruoyu et al. (2024). Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — arXiv (Cornell University). Full text read. Flagship production evidence for KV-cache-centric disaggregation and early-rejection scheduling.doi:10.48550/arxiv.2407.00079
Hu, Cunchen et al. (2024). MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool — arXiv (Cornell University). Full text read. Shows the compounding benefit of unifying context caching with disaggregated inference via an elastic distributed memory pool.doi:10.48550/arxiv.2406.17565
Sheng, Ying et al. (2023). FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — arXiv (Cornell University). Full text read. Shows LLM-scale serving on commodity single-GPU hardware via linear-programming-guided tensor placement and offloading.doi:10.48550/arxiv.2303.06865
Pope, Reiner et al. (2022). Efficiently Scaling Transformer Inference — arXiv (Cornell University). Full text read. Establishes the latency-vs-MFU Pareto framing and the KV-memory/context-length tradeoff for giant models.doi:10.48550/arxiv.2211.05102
Wu, Bingyang et al. (2023). Fast Distributed Inference Serving for Large Language Models — arXiv (Cornell University). Full text read. First preemptive (token-granularity) LLM serving scheduler targeting head-of-line blocking in interactive inference.doi:10.48550/arxiv.2305.05920
Sheng, Ying et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters — arXiv (Cornell University). Full text read. Shows memory-pooling (unified paging) generalizes from KV cache to adapter weights for multi-tenant fine-tuned serving.doi:10.48550/arxiv.2311.03285
Zhenyu, Zhang, et al. (2023). H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — arXiv (Cornell University). Full text read. Landmark KV eviction paper; introduces heavy-hitter oracle and submodular formulation of eviction.doi:10.48550/arxiv.2306.14048
Liu, Zichang et al. (2023). Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time — arXiv (Cornell University). Full text read. Early training-free KV eviction evidence linking attention persistence to cache budget reduction.doi:10.48550/arxiv.2305.17118
Li, Yuhong et al. (2024). SnapKV: LLM Knows What You are Looking for Before Generation — arXiv (Cornell University). Full text read. Training-free prompt KV compression by per-head important-position selection; strong for long-context memory and latency reduction.doi:10.48550/arxiv.2404.14469
Xiao, Guangxuan et al. (2023). Efficient Streaming Language Models with Attention Sinks — arXiv preprint. Full text read. Attention-sink insight underpins KV eviction policies (sink+recent) used across streaming, drafting, and cache-compression systems.doi:10.48550/arxiv.2309.17453
Liu, Zirui et al. (2024). KIVI : Plug-and-play 2bit KV Cache Quantization with Streaming Asymmetric Quantization — arXiv preprint (ICLR 2024). Full text read. Landmark evidence that KV cache tolerates 2-bit quantization with per-channel/per-token asymmetry, directly attacking the decode memory bottleneck.doi:10.48550/arxiv.2402.02750
Hooper, Coleman et al. (2024). KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization — arXiv (Cornell University). Full text read. Sub-4-bit KV cache quantization suite (per-channel, pre-RoPE, non-uniform, dense-and-sparse) that makes million-token contexts feasible on one GPU.doi:10.48550/arxiv.2401.18079
Kang, Hao et al. (2024). GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM — arXiv (Cornell University). Full text read. Shows hybrid quantization+low-rank+sparse beats uniform quantization/eviction for high-ratio KV compression.doi:10.48550/arxiv.2403.05527
Gim, In et al. (2023). Prompt Cache: Modular Attention Reuse for Low-Latency Inference — arXiv (Cornell University). Full text read. Early evidence that cross-request KV reuse (precomputed attention states) accelerates prompt-heavy workloads.doi:10.48550/arxiv.2311.04934
Yao, Jiayi et al. (2024). CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — arXiv (Cornell University). Full text read. Enables RAG-style chunk reuse beyond prefix caching, a key building block for cache-centric serving.doi:10.48550/arxiv.2405.16444
Jin, Chao et al. (2025). RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation — ACM Transactions on Computer Systems. Abstract only. Shows KV caching across the memory hierarchy plus retrieval-generation overlap as levers for RAG serving latency.doi:10.1145/3768628
Liu, Yuhan et al. (2024). CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Proceedings of the ACM SIGCOMM 2024 Conference. Abstract only. Lossy-but-tuned KV cache encoding that attacks context-loading network delay in long-context serving.doi:10.1145/3651890.3672274
Frantar, Elias et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — arXiv (Cornell University). Full text read. Foundational one-shot PTQ result establishing 4-bit weights as the practical default for LLM deployment.doi:10.48550/arxiv.2210.17323
Lin, Ji et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — arXiv (Cornell University). Full text read. Hardware-friendly weight-only PTQ that became a standard on-device 4-bit serving stack; complements KV-cache and activation quantization.doi:10.48550/arxiv.2306.00978
Xiao, Guangxuan et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models — arXiv (Cornell University). Full text read. Foundational W8A8 PTQ that halves memory and nearly doubles GEMM throughput; enabler of single-node 500B+ serving.doi:10.48550/arxiv.2211.10438
Dettmers, Tim et al. (2022). LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale — Advances in Neural Information Processing Systems 35. Abstract only. Foundational outlier-aware 8-bit quantization that made large-model inference accessible on commodity hardware.doi:10.52202/068431-2198
Yao, Zhewei et al. (2022). ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers — arXiv (Cornell University). Full text read. Early full-stack W8A8 evidence, including a data-free distillation trick, relevant to activation-quantization history.doi:10.48550/arxiv.2206.01861
Chee, Jerry et al. (2023). QuIP: 2-Bit Quantization of Large Language Models With Guarantees — arXiv preprint. Full text read. Theoretical anchor for why incoherence enables extreme low-bit weight quantization, informing later QuIP# and 2-bit methods.doi:10.48550/arxiv.2307.13304
Tseng, Albert et al. (2024). QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks — arXiv preprint. Full text read. Advances the 2-bit frontier for weight-only PTQ via lattice codebooks; complements GPTQ-style methods in the review.doi:10.48550/arxiv.2402.04396
Liu, Zechun et al. (2024). SpinQuant: LLM quantization with learned rotations — arXiv (Cornell University). Full text read. Evidence that learned (not random) rotation matrices are key to accurate full-4-bit quantization including KV cache.doi:10.48550/arxiv.2405.16406
Micikevicius, Paulius et al. (2022). FP8 Formats for Deep Learning — arXiv preprint. Full text read. Defines the FP8 standard that underpins modern 8-bit inference engines — foundational for the quantization chapter.doi:10.48550/arxiv.2209.05433
Lin, Yujun et al. (2024). QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving — arXiv (Cornell University). Full text read. Key evidence that W4A8KV4 with dequantization-aware kernels unlocks INT4 speedups in large-batch cloud serving, not just edge.doi:10.48550/arxiv.2405.04532
Zhao, Yilong et al. (2023). Atom: Low-bit Quantization for Efficient and Accurate LLM Serving — arXiv (Cornell University). Full text read. Shows low-bit weight-activation quantization translating directly into serving throughput via INT4 tensor cores.doi:10.48550/arxiv.2310.19102
Shao, Wenqi et al. (2023). OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models — arXiv (Cornell University). Full text read. Omnidirectional PTQ covering extreme low-bit weight-activation regimes at QAT-like quality with PTQ-level cost.doi:10.48550/arxiv.2308.13137
Kim, Sehoon et al. (2023). SqueezeLLM: Dense-and-Sparse Quantization — arXiv (Cornell University). Full text read. Shows 3-bit weight quantization can be lossless-ish with non-uniform sensitivity-aware schemes.doi:10.48550/arxiv.2306.07629
Dettmers, Tim et al. (2023). SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression — arXiv (Cornell University). Full text read. Early near-lossless 3-4 bit weight quantization enabling consumer-GPU deployment; baseline for later outlier-aware PTQ methods.doi:10.48550/arxiv.2306.03078
Dettmers, Tim & Zettlemoyer, Luke (2022). The case for 4-bit precision: k-bit Inference Scaling Laws — arXiv preprint. Full text read. Theoretical anchor for the field's 4-bit default in weight quantization.doi:10.48550/arxiv.2212.09720
Wang, Hongyu et al. (2023). BitNet: Scaling 1-bit Transformers for Large Language Models — arXiv preprint. Full text read. Evidence for 1-bit weights as an end-goal architecture; motivates extreme low-bit inference research.doi:10.48550/arxiv.2310.11453
Chen, Charlie et al. (2023). Accelerating Large Language Model Decoding with Speculative Sampling — arXiv (Cornell University). Full text read. Foundational speculative-decoding result: parallel verification makes drafting an exact, distribution-preserving speedup.doi:10.48550/arxiv.2302.01318
Leviathan, Yaniv et al. (2022). Fast Inference from Transformers via Speculative Decoding — arXiv (Cornell University). Full text read. Foundational speculative-decoding paper: formal rejection sampling guaranteeing the target distribution, plus speedup analysis.doi:10.48550/arxiv.2211.17192
Cai, Tianle et al. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — arXiv (Cornell University). Full text read. Draft-model-free parallel decoding (multi-token heads + tree verification); benchmark anchor for tree-based speculative methods.doi:10.48550/arxiv.2401.10774
Li, Yuhui et al. (2024). EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — arXiv preprint. Full text read. State-of-the-art draft-based speculation: predicting at the feature level yields higher acceptance than token-level drafts.doi:10.48550/arxiv.2401.15077
Li, Yuhui et al. (2024). EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees — arXiv preprint. Full text read. State-of-the-art lossless speculative decoding: confidence-guided dynamic draft trees beat static trees.doi:10.48550/arxiv.2406.16858
Stern, Mitchell et al. (2018). Blockwise Parallel Decoding for Deep Autoregressive Models — arXiv preprint. Full text read. Theoretical origin of blockwise speculative decoding — canonical citation for the speculative-decoding chapter.doi:10.48550/arxiv.1811.03115
He, Zhenyu et al. (2024). REST: Retrieval-Based Speculative Decoding — Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human. Abstract only. Training-free speculative decoding that drafts from retrieved n-grams instead of a learned draft model.doi:10.18653/v1/2024.naacl-long.88
Miao, Xupeng et al. (2023). SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification — arXiv (Cornell University). Full text read. Shows token-tree speculation generalizes speculative decoding to multi-token parallel verification in serving systems.doi:10.48550/arxiv.2305.09781
Spector, Benjamin & Ré, Chris (2023). Accelerating LLM Inference with Staged Speculative Decoding — arXiv (Cornell University). Full text read. Evidence that speculative decoding targets the low-arithmetic-intensity, small-batch on-device regime.doi:10.48550/arxiv.2308.04623
Chen, Ziyi et al. (2023). Cascade Speculative Drafting for Even Faster LLM Inference — arXiv preprint. Full text read. Pushes speculative drafting beyond autoregressive draft models via cascaded statistical drafting.doi:10.48550/arxiv.2312.11462
Sun, Hanshi et al. (2024). TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding — arXiv preprint. Full text read. Long-context speculative decoding that exploits attention sparsity for drafting, closing the gap between on-chip and offloaded long-sequence serving.doi:10.48550/arxiv.2404.11912
Lepikhin, Dmitry et al. (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — arXiv preprint. Full text read. Foundational evidence that MoE scales quality sublinearly in compute cost — background for sparse-inference arguments.doi:10.48550/arxiv.2006.16668
Fedus, William et al. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — arXiv (Cornell University). Full text read. Canonical sparse MoE scaling result; motivates expert-parallel serving systems and inference-aware MoE design.doi:10.48550/arxiv.2101.03961
Hwang, Chang Ho et al. (2022). Tutel: Adaptive Mixture-of-Experts at Scale — arXiv (Cornell University). Full text read. Evidence that adaptive MoE parallelism matters for scaling expert-parallel inference; relevant to MoE serving background.doi:10.48550/arxiv.2206.03382
Rajbhandari, Samyam et al. (2022). DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale — arXiv (Cornell University). Full text read. Early evidence that MoE + expert pruning/quantization yields large latency and cost wins over dense models.doi:10.48550/arxiv.2201.05596
Jiang, Albert Q. et al. (2024). Mixtral of Experts — arXiv (Cornell University). Full text read. Reference MoE model: sparse activation reduces per-token compute but keeps full parameter memory footprint.doi:10.48550/arxiv.2401.04088
DeepSeek-AI et al. (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — arXiv preprint. Full text read. Production-scale evidence that latent KV compression (MLA) plus MoE sparsity radically cuts cache and raises serving throughput.doi:10.48550/arxiv.2405.04434
DeepSeek-AI et al. (2024). DeepSeek-V3 Technical Report — arXiv preprint. Full text read. Production-scale evidence that MLA + MoE architectures cut KV cache and active compute, shaping the inference-optimization design space.doi:10.48550/arxiv.2412.19437
Liu, Hao et al. (2023). Ring Attention with Blockwise Transformers for Near-Infinite Context — arXiv (Cornell University). Full text read. Distributed long-context evidence: KV communication can be hidden behind blockwise attention compute.doi:10.48550/arxiv.2310.01889
Yang, Amy et al. (2024). Context Parallelism for Scalable Million-Token Inference — arXiv preprint. Full text read. Lossless pass-KV/pass-Q ring attention variants make million-token prefill latency practical and are compatible with KV-cache quantization and paging.doi:10.48550/arxiv.2411.01783
Shazeer, Noam (2019). Fast Transformer Decoding: One Write-Head is All You Need — arXiv preprint. Full text read. Origin of MQA, the KV-head-sharing idea that underlies GQA and modern serving-efficient LLMs (memory-bandwidth-bound decode analysis).doi:10.48550/arxiv.1911.02150
Ainslie, Joshua et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Abstract only. Establishes GQA as the de-facto KV-cache-reducing attention variant used by nearly all later inference-optimized LLMs.doi:10.18653/v1/2023.emnlp-main.298
Gu, Albert & Dao, Tri (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces — arXiv (Cornell University). Full text read. Evidence that selective SSM architectures remove the KV cache bottleneck entirely, a structural alternative to cache optimization.doi:10.48550/arxiv.2312.00752
Song, Yixin et al. (2024). PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU — Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. Abstract only. Demonstrates activation sparsity as a lever for single-GPU/PC-class LLM inference without accuracy loss.doi:10.1145/3694715.3695964
Zhou, Zixuan et al. (2024). A Survey on Efficient Inference for Large Language Models — arXiv (Cornell University). Full text read. Quantitative anchor points (memory, latency, operator breakdown) for the review's problem-motivation section.doi:10.48550/arxiv.2404.14294
Wan, Zhongwei et al. (2023). Efficient Large Language Models: A Survey — arXiv preprint. Full text read. Broad taxonomy useful for structuring the review's model-compression vs system sections.doi:10.48550/arxiv.2312.03863
Zhu, Xunyu et al. (2023). A Survey on Model Compression for Large Language Models — arXiv (Cornell University). Full text read. Useful map of quantization/pruning/KD landscape and evaluation metrics for the compression side of inference optimization.doi:10.48550/arxiv.2308.07633
Chitty-Venkata, Krishna Teja et al. (2023). A survey of techniques for optimizing transformer inference — Journal of Systems Architecture. Abstract only. Broad background taxonomy for the review's algorithmic-level (quantization/pruning/distillation) and accelerator sections.doi:10.1016/j.sysarc.2023.102990
Yuhan, Liu, et al. (2025). LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — arXiv preprint. Full text read. Open-source KV cache offloading/sharing layer enabling cross-engine prefix reuse and PD disaggregation; key infrastructure for cache-centric serving.doi:10.48550/arxiv.2510.09665
Zefan, Cai, et al. (2024). PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling — arXiv preprint. Full text read. Layer-wise KV budget allocation as a memory-saving strategy distinct from uniform eviction.doi:10.48550/arxiv.2406.02069
Yichao, Fu, et al. (2024). Break the Sequential Dependency of LLM Inference Using Lookahead Decoding — arXiv preprint. Full text read. Draft-model-free alternative to speculative decoding, relevant for deployments where drafts are unavailable.doi:10.48550/arxiv.2402.02057
Lin, Bin et al. (2024). Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache — arXiv (Cornell University). Full text read. Shows layer-level disaggregation plus pooled memory as a route to elastic long-context serving.doi:10.48550/arxiv.2401.02669
Yunfei, Cheng, et al. (2024). Recurrent Drafter for Fast Speculative Decoding in Large Language Models — arXiv preprint. Full text read. State-of-the-art speculative decoding via recurrent drafting; strong H100 and on-device numbers.doi:10.48550/arxiv.2403.09919
Wei, Xiuying et al. (2023). Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling — Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Abstract only. Equivalent shifting/scaling tames asymmetric activation outliers, a key enabler for low-bit PTQ of LLMs.doi:10.18653/v1/2023.emnlp-main.102
Li, Yuhui et al. (2025). EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Advances in Neural Information Processing Systems 38. Abstract only. Scales speculative drafting via direct token prediction and multi-layer feature fusion, showing speedups improve with more training data.doi:10.52202/085713-4562
Agrawal, Amey et al. (2024). Vidur: A Large-Scale Simulation Framework For LLM Inference — arXiv (Cornell University). Full text read. Enables cheap configuration search (batching/scheduling/parallelism) without expensive GPU experiments; key methodology for the review.doi:10.48550/arxiv.2405.05465
Wang, Yuxin et al. (2025). BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems — Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. Abstract only. Provides the realistic workload grounding needed to evaluate KV cache, scheduling, and disaggregation optimizations beyond synthetic assumptions.doi:10.1145/3711896.3737413
Yuan, Zhihang et al. (2024). LLM Inference Unveiled: Survey and Roofline Model Insights — arXiv (Cornell University). Full text read. Provides the roofline framing that organizes the review's discussion of memory-bound decode vs compute-bound prefill.doi:10.48550/arxiv.2402.16363
Li, Baolin et al. (2024). LLM Inference Serving: Survey of Recent Advances and Opportunities — arXiv (Cornell University). Full text read. Map of the serving-systems landscape (batching, disaggregation, scheduling) for the review's system-level section.doi:10.48550/arxiv.2407.12391
Hu, Cunchen et al. (2024). Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads — arXiv (Cornell University). Full text read. Quantifies prefill-decode interference and validates chunked prefill + prefill/decode disaggregation plus two-level scheduling.doi:10.48550/arxiv.2401.11181
Forys, Przemyslaw et al. (2026). When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference — arXiv (Cornell University). Full text read. Forward-looking simulation evidence that heterogeneous/disaggregated hardware, not just software, is needed for agentic inference.doi:10.48550/arxiv.2608.03741
Wang, Chao et al. (2025). Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving — arXiv (Cornell University). Full text read. Resolves the aggregation-vs-disaggregation debate: optimal regime depends on SLO mix, and hybrid modes win under balanced SLOs.doi:10.48550/arxiv.2508.01989
Gao, Lei et al. (2025). DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing — arXiv (Cornell University). Full text read. Adaptive middle ground between phase aggregation and disaggregation for the serving-systems chapter.doi:10.48550/arxiv.2511.04791
Li, Cong et al. (2026). Tetris: Efficient Long-context LLM Serving with Chunkwise Dynamic Sequence Parallelism — 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA). Abstract only. Fine-grained intra-request sequence-parallelism scheduling that exploits fragmented resources in disaggregated serving.doi:10.1109/isca66397.2026.00098
Lai, Ruiqi et al. (2025). TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocity — arXiv (Cornell University). Full text read. Evidence that disaggregated serving needs proactive, token-level autoscaling rather than lagging utilization metrics.doi:10.48550/arxiv.2512.03416
Li, Zongze et al. (2026). Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving — arXiv (Cornell University). Full text read. Quantifies append-prefill vs full-prefill interference, informing PD-disaggregation design choices.doi:10.48550/arxiv.2603.13358
She, Jianshu et al. (2026). LAPS: A Length-Aware-Prefill LLM Serving System — arXiv (Cornell University). Full text read. Extends PD disaggregation with length-aware prefill scheduling (long/short prefill separation, dual-queue, CUDA-Graph batching).doi:10.48550/arxiv.2601.11589
Ye, Zihao et al. (2025). FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving — arXiv (Cornell University). Full text read. Evidence that attention-kernel engineering (block-sparse KV formats, JIT) delivers large decode-latency wins across serving stacks.doi:10.48550/arxiv.2501.01005
Lee, Wonbeom et al. (2024). InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — arXiv (Cornell University). Full text read. Selective KV prefetch as a bridge between offloading and cache management for long contexts.doi:10.48550/arxiv.2406.19707
Strati, Foteini et al. (2024). DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving — arXiv (Cornell University). Full text read. KV cache streaming as a unifying primitive for disaggregation, memory management, and fault tolerance in distributed serving.doi:10.48550/arxiv.2403.01876
Liu, Yuhan et al. (2024). DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving — arXiv (Cornell University). Full text read. First cross-model KV cache reuse system (selective layer recomputation + pipelined loading) for compound/agentic multi-LLM serving.doi:10.48550/arxiv.2411.02820
Chen, Shiyang et al. (2024). KVDirect: Distributed Disaggregated LLM Inference — arXiv (Cornell University). Full text read. Key evidence that prefill-decode disaggregation can be extended beyond a single node, addressing the scalability limit of DistServe-style designs.doi:10.48550/arxiv.2501.14743
Pengju, Yang (2026). SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Serving — arXiv preprint. Full text read. Argues PD KV transfer should be treated as a precision-allocation problem rather than binary token selection.doi:10.48550/arxiv.2606.08635
Qiu, Haoran et al. (2024). Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction — arXiv (Cornell University). Full text read. Length-aware scheduling addresses head-of-line blocking from autoregressive length variability.doi:10.48550/arxiv.2404.08509
Zheng, Zangwei et al. (2023). Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline — arXiv (Cornell University). Full text read. Early demonstration that output-length prediction enables smarter batching — precursor to SJF schedulers.doi:10.48550/arxiv.2305.13144
Xiao, Guangxuan et al. (2024). DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads — arXiv (Cornell University). Full text read. Strong evidence that head-level specialization (retrieval vs streaming) makes long-context serving dramatically cheaper.doi:10.48550/arxiv.2410.10819
Gu, Xuejing et al. (2024). When Attention Sink Emerges in Language Models: An Empirical View — arXiv (Cornell University). Full text read. Theoretical grounding for attention-sink-based optimizations (StreamingLLM-style KV tricks, quantization).doi:10.48550/arxiv.2410.10781
Ribar, Luka et al. (2023). SparQ Attention: Bandwidth-Efficient LLM Inference — arXiv (Cornell University). Full text read. Demonstrates bandwidth-side KV optimization: predicting high-attention tokens can yield large transfer savings in decode.doi:10.48550/arxiv.2312.04985
Ge, Suyu et al. (2023). Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs — arXiv (Cornell University). Full text read. Early evidence that head-specific KV eviction policies can cut memory with negligible quality loss, precursor to retrieval-head methods.doi:10.48550/arxiv.2310.01801
Chen, Yilong et al. (2024). NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time — Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Abstract only. Single-pass encoding-phase KV eviction framework showing cache can be halved with near-full task performance.doi:10.18653/v1/2024.acl-long.428
Eliseev, Artyom & Mazur, D. Peter (2023). Fast Inference of Mixture-of-Experts Language Models with Offloading — arXiv (Cornell University). Full text read. Shows MoE expert offloading + LRU caching makes large MoE models usable on commodity hardware.doi:10.48550/arxiv.2312.17238
Shen, Haihao et al. (2025). Efficient LLM Inference on CPUs — Machine Translation: Technologies and Applications. Full text read. Evidence that weight-only INT4 plus AMX kernels makes LLM inference viable on commodity CPUs, relevant to the hardware-aware quantization thread.doi:10.1007/978-3-031-85747-8_3
Xu, Daliang et al. (2023). LLMCad: Fast and Scalable On-device Large Language Model Inference — arXiv (Cornell University). Full text read. On-device speculative decoding via token trees with self-adjusting fallback and compute-IO pipelining; shows speculative methods can break the mobile memory wall.doi:10.48550/arxiv.2309.04255
Alizadeh, Keivan et al. (2024). LLM in a flash: Efficient Large Language Model Inference with Limited Memory — Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Abstract only. Hardware-informed flash/DRAM orchestration that makes very large models feasible on memory-limited edge devices.doi:10.18653/v1/2024.acl-long.678
Yang, Dongjie et al. (2024). PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference — Findings of the Association for Computational Linguistics ACL 2024. Abstract only. Layer-wise crucial-context retention compresses KV cache during pre-computation, cutting both memory and compute.doi:10.18653/v1/2024.findings-acl.195
Wu, Hao‐Yi & Tu, Kewei (2024). Layer-Condensed KV Cache for Efficient Inference of Large Language Models — Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Abstract only. Orthogonal to other memory-saving techniques; drastic throughput gains from caching KVs at a subset of layers.doi:10.18653/v1/2024.acl-long.602
Ye, Lu et al. (2024). ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition — Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Abstract only. Prefix-aware KV cache sharing converts redundant system-prompt compute into memory and kernel-level wins.doi:10.18653/v1/2024.acl-long.623
Yuan, Jiayi et al. (2024). KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches — Findings of the Association for Computational Linguistics: EMNLP 2024. Abstract only. First reasonably aligned head-to-head benchmark of long-context efficiency methods, useful for situating KV compression claims.doi:10.18653/v1/2024.findings-emnlp.266
Gong, Ruihao et al. (2024). LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit — Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. Abstract only. Standardized plug-and-play toolkit enabling fair comparison of LLM quantization methods in a fragmented literature.doi:10.18653/v1/2024.emnlp-industry.12
Xia, Heming et al. (2024). Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding — Findings of the Association for Computational Linguistics ACL 2024. Abstract only. Reference survey structuring the speculative decoding landscape for the review's decoding-acceleration chapter.doi:10.18653/v1/2024.findings-acl.456
Elhoushi, Mostafa et al. (2024). LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding — Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Abstract only. Training recipe plus self-speculative decoding that reuses the same model as drafter and verifier with shared activations.doi:10.18653/v1/2024.acl-long.681
Svirschevski, Ruslan et al. (2024). SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices — Advances in Neural Information Processing Systems 37. Full text read. Demonstrates speculative execution's largest gains in bandwidth-bound offloaded settings, extending speculation to consumer hardware.doi:10.52202/079017-0522
Li, Minghan et al. (2024). Nearest Neighbor Speculative Decoding for LLM Generation and Attribution — Advances in Neural Information Processing Systems 37. Full text read. Speculative decoding meets semi-parametric retrieval — speed plus attribution for knowledge-intensive generation.doi:10.52202/079017-2574
Zhong, Shuzhang et al. (2024). ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decoding — Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design. Abstract only. Improves tree-based parallel decoding by pruning unpromising branches and adapting tree shape to batch size and task.doi:10.1145/3676536.3676695
Park, Daon & Egger, Bernhard (2024). Improving Throughput-oriented LLM Inference with CPU Computations — Proceedings of the 2024 International Conference on Parallel Architectures and Compilation Techniques. Abstract only. Evidence that harvesting idle CPU compute alongside GPU offloading can nearly double commodity-hardware throughput.doi:10.1145/3656019.3676949
Dai, Damai et al. (2024). DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Abstract only. Architectural template (fine-grained + shared experts) later deployed in DeepSeek-V2/V3 serving systems.doi:10.18653/v1/2024.acl-long.70
Yao, Jinghan et al. (2024). Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference — 2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Abstract only. Shows expert placement exploiting inter-layer affinity can halve MoE all-to-all communication overhead.doi:10.1109/ipdps57955.2024.00086
Ma, Ruilong et al. (2024). HPipe: Large Language Model Pipeline Parallelism for Long Context on Heterogeneous Cost-effective Devices — Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track). Abstract only. Pipeline parallelism adapted to heterogeneous low-cost hardware for long-context private deployment.doi:10.18653/v1/2024.naacl-industry.1
Fang, Jiarui & Zhao, Shangchun (2024). USP: A Unified Sequence Parallelism Approach for Long Context Generative AI — arXiv (Cornell University). Full text read. Shows sequence-parallelism design space (Ulysses vs Ring vs hybrid) for long-context compute.doi:10.48550/arxiv.2405.07719
Du, Nan et al. (2021). GLaM: Efficient Scaling of Language Models with Mixture-of-Experts — arXiv (Cornell University). Full text read. Early scaling evidence that sparse MoE cuts per-token inference compute at constant quality.doi:10.48550/arxiv.2112.06905
Su, Qidong et al. (2023). The Synergy of Speculative Decoding and Batching in Serving Large Language Models — arXiv (Cornell University). Full text read. Evidence that batching and speculative decoding interact, so speculation length must be batch-adaptive.doi:10.48550/arxiv.2310.18813
Zhang, Jun et al. (2024). Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding — Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Abstract only. Lossless, training-free, zero-extra-memory self-speculative decoding via layer skipping.doi:10.18653/v1/2024.acl-long.607
Kim, Sehoon et al. (2023). Speculative Decoding with Big Little Decoder — arXiv (Cornell University). Full text read. Big-little speculative decoding framework — early evidence that draft-then-verify accelerates decoding.doi:10.48550/arxiv.2302.07863