Tag: benchmarking
2 reviews
Synthetic workload generation for benchmarking LLM inference
Which open-source tools and datasets can generate synthetic workloads — chat, agentic, and other — for benchmarking LLM inference performance?
A practical map of the tools that generate workloads for benchmarking LLM inference: engine-native load generators (vLLM, SGLang, llama.cpp), trace-driven generators (BurstGPT, Azure traces, ShareGPT/LMSYS replay), statistical generators calibrated to production traffic (ServeGen, FineServe), simulators (LLMServingSim, Frontier, SimLLM), and the emerging agentic-workload tooling (XPerf, SGLang agentic-trace, GuideLLM tool calling). The evidence says workload realism matters — Poisson arrivals and fixed token lengths misrepresent production traffic — and that agentic workloads are the least standardized but fastest-moving category. Confidence is moderate: repo capabilities were verified from primary docs fetched this session, but several tools change weekly and one major tool (NVIDIA GenAI-Perf) was unretrievable.
Benchmarking tokens per watt: how AI inference energy efficiency is measured
What benchmarks and measurement methodologies exist for quantifying AI inference energy efficiency in tokens per watt (or per joule), from accelerator to data-centre level, and what do they establish?
The literature on measuring AI inference efficiency in tokens-per-watt terms is young (mostly 2023-2026) and fragmented: one consortium standard exists at the system level (MLPerf Power), but the benchmarks that actually report tokens per watt or joules per token are research tools with incompatible measurement boundaries. Measured numbers span orders of magnitude — roughly 3-4 joules per output token for a 65B model on A100s, 0.002 to 2.9 kWh per 1,000 inferences depending on task, a 65x spread across models in commercial data centres, and a proposed 1/W law under which tokens per watt halves each time the context window doubles. No retrieved benchmark measures tokens per watt at the data-centre (facility) level; facility efficiency is still expressed as PUE, so wall-level tokens per watt is derived, not measured. Confidence is moderate: 20 of 65 sources were read in full text and several prominent items were unreachable in-session.