Tag: vLLM

The LLM inference engine playbook: tricks in vLLM and SGLang

Which optimization techniques do the vLLM and SGLang serving engines implement, and in what order should an engineer study them?

LLM serving engines are built from a stack of optimization tricks, each attacking one bottleneck: batching keeps the GPU busy, KV-cache paging and reuse stop memory from fragmenting or being recomputed, faster attention kernels and lower-precision arithmetic cut bytes moved, speculative decoding trades spare compute for fewer serial steps, and prefill-decode disaggregation lets two workloads with opposite resource profiles run on machines tuned for each. This review catalogs the trick list of the two dominant open-source engines, vLLM and SGLang, maps each trick to the paper that introduced it and the bottleneck it addresses, and closes with an ordered self-study path. Confidence is moderate: the mechanisms are well established in the literature, but the engines' own performance claims are vendor-reported and rarely independently benchmarked.

LLM inference · vLLM · SGLang · serving systems · KV cache · speculative decoding · prefill-decode disaggregation · self-study guide