Tag: mixture-of-experts

Optimizing large language model inference

Which techniques most improve the throughput, latency, and cost of large language model inference?

Large language model inference is memory-bound, not compute-bound, and every major optimization family attacks the memory wall from a different angle: batching and scheduling to keep GPUs busy, KV-cache management to shrink the state that grows with context, quantization to cut bytes per weight, speculative decoding to trade spare compute for fewer serial steps, and prefill-decode separation to stop two workloads with opposite resource profiles from interfering. The evidence is consistent that these techniques deliver large gains — commonly 2–6x in throughput or latency, occasionally 10x or more in narrow regimes — but nearly every headline number is an "up to" figure measured on the authors' own hardware and workload, and independent benchmarks are only beginning to appear.

LLM inference · serving systems · KV cache · quantization · speculative decoding · mixture-of-experts · prefill-decode disaggregation