Synthetic workload generation for benchmarking LLM inference

Why this question

Anyone operating or building LLM inference infrastructure needs to know how a serving stack behaves before trusting it in production — and a benchmark is only as good as its workload. Benchmarks built on naive synthetic traffic (fixed-length prompts, Poisson arrivals) systematically miss the effects that dominate real deployments: request bursts that overflow queues, long-tail token lengths that exhaust KV cache, multi-turn sessions that defeat prefix-cache reuse, and agentic execution graphs that turn one user request into dozens of dependent LLM calls ²²²⁴.

The question matters for a second reason: the tooling is fragmenting quickly. Serving engines ship their own benchmark scripts, MLCommons standardizes load generation for compliance-style runs, academic groups release trace datasets, and a new wave of agentic-workload benchmarks is arriving monthly. Choosing wrong means either benchmarking an unrepresentative workload (and tuning the wrong knob) or benchmarking the wrong layer entirely. This review maps the landscape so a practitioner can pick a generator per workload type — chat, agentic, RAG, batch, long-context — rather than improvising.

Scope and methods

Question. Which open-source tools and datasets can generate synthetic workloads — chat, agentic, and other — for benchmarking LLM inference performance?

Inclusion criteria. Open-source or publicly documented tools whose primary purpose is generating or replaying LLM serving workloads (load generators, trace datasets, workload simulators, agentic task suites); papers characterizing LLM serving workloads or introducing the benchmarks; anything published through September 2026.

Exclusion criteria. Model-quality evaluation suites (e.g., promptfoo, lm-eval-harness, HELM) unless they double as load generators; training-focused simulation; commercial benchmarking services unless open-sourced; workload tools whose repos or docs could not be retrieved in-session.

Screening counts. GitHub Search API: 8 discovery queries (~9,600 candidate repos returned) plus ~30 exact-name checks → ~30 shortlisted → 23 repos read from primary docs. arXiv: 42 query strings → ~30 candidate papers → 26 included; dropped at screening: a wrong-title DistServe match, a “Serving LLM Systems” survey that does not exist on arXiv, a Splitwise false positive, and one dataset paper whose cited DOI 404s (ACDC). PyPI and doi.org used for artifact and DOI verification. Every DOI in sources.json was verified to resolve this session; full author lists came from arXiv API records.

Evidence model. Two layers, as in prior tool-focused reviews on this site: the implementation layer — what each tool does today — comes from each repo’s README and official docs fetched in-session and is attributed to the vendor (“per the repository README”). The mechanism layer — why workload realism matters, what real traffic looks like — comes from the 26 cited papers. The repos themselves carry no DOI and so cannot be sources.json entries; they are linked inline in the body. The review deliberately does not benchmark the tools against each other (that would require the hardware); it reports capabilities as documented.

The landscape

The field is young and layered. Three generations of tooling coexist. First, engine-native load generators (2019–2024): the benchmark scripts that ship inside vLLM, SGLang, llama.cpp, and TensorRT-LLM, descended from FastChat’s ShareGPT-bench and MLPerf’s LoadGen scenario model ¹⁷¹². Second, trace datasets and calibrated generators (2023–2026): BurstGPT, Azure’s LLM inference traces, LMSYS-Chat-1M, and — the current research frontier — ServeGen and FineServe, which model arrival and length statistics from production traffic instead of assuming them ^9102225. Third, agentic workload tooling (2024–2026): XPerf’s trace-replay framework, SGLang’s agentic-trace dataset mode, GuideLLM’s tool-calling profiles, and agent-task suites (tau-bench, AgentBench, TheAgentCompany, WebArena, ToolLLM, GAIA) that generate the multi-turn, tool-heavy request chains agents produce ^237819202118.

In parallel, a simulation branch (LLMServingSim, Frontier, SimLLM, Simthesizer) replaces the GPU with a calibrated model so workload × scheduler × hardware interactions can be studied without hardware ¹⁵¹⁶²⁶. And a content-generation branch (Self-Instruct, Evol-Instruct, AgentInstruct, distilabel, RAGAS) generates the text of workloads — prompts and tasks — rather than the traffic around them ^11121314.

Two shape statements: the academic workload-characterization literature is dominated by a handful of trace sources (Azure, Alibaba, one global marketplace), and the agentic-serving literature is dominated by 2025–2026 preprints — a sign the category is real but not yet standardized.

Theme 1 — Real traffic is bursty, non-stationary, and workload-specific

The load-bearing finding across every trace study is that naive traffic models misrepresent production. BurstGPT, a 121-day trace of Azure-hosted ChatGPT/GPT-4 serving (~5.3M requests per release), shows pronounced burstiness, weekly and daily periodicity, and distinct conversation-vs-API modes ⁹. ServeGen, built from billions of requests across 12 Alibaba production models, quantifies it: request arrivals exhibit “a complex bursty pattern that goes beyond any single stochastic process,” with coefficients of variation above 1, so Poisson arrivals (CV = 1) fit poorly while Gamma and Weibull fits vary by client — and input/output length distributions shift over days and weeks ²². FineServe, a multi-model trace from a global commercial marketplace, reaches the same conclusion from a different population: non-stationary, bursty arrivals that differ by model architecture and task intent (writing, coding, reasoning, role-playing) ²⁵.

The serving-systems literature built on these traces shows why the details matter. Splitwise found real chat traffic is phase-asymmetric — prefill and decode strain different resources — which naive fixed-length benchmarks cannot expose ⁴. DistServe’s goodput gains under realistic SLOs depend on realistic workload mixes ⁵. FastServe’s scheduling results hold for interactive, multi-turn workloads specifically ⁶. The LLM-serving survey frames workload characterization as one of the field’s open pillars ³. Practical implication: a benchmark that fixes input/output lengths and uses Poisson arrivals is benchmarking a workload that does not exist in production.

Theme 2 — Engine-native load generators: the practical default for chat

Every major open serving engine now ships a load generator with tunable synthetic and dataset-driven workloads. These are the fastest path to a first benchmark.

vLLM consolidated its scripts into a CLI, vllm bench serve (with bench throughput, bench latency, and bench sweep serve variants). Per the current vLLM benchmarking docs, its dataset registry spans ShareGPT, ShareGPT4V/Video, BurstGPT traces (downloaded directly from the BurstGPT release), a fully synthetic Random dataset, Prefix Repetition, multimodal synthetic data, function-calling BFCL v3 categories, Spec-Bench, SPEED-Bench, HuggingFace datasets (MT-Bench, HumanEval, GSM8K), and custom JSONL. Arrival modeling is the interesting part: --request-rate defaults to inf (burst), and finite rates use a Poisson process whose shape is controlled by --burstiness (1.0 = natural Poisson, 0.1–0.5 = bursty, 2.0–5.0 = uniform spacing); --max-concurrency caps in-flight requests. The docs explicitly warn that repeated runs can reuse prefix cache and inflate throughput, and recommend varying --seed or using bench sweep ¹.

SGLang’s python -m sglang.bench_serving supports the same class of workloads with different coverage: sharegpt (default), random (lengths sampled from ShareGPT’s token space), random-ids, image, generated-shared-prefix (synthetic long system prompts with zipf-distributed prefix-group sampling), mmmu, speed-bench (speculative-decoding evaluation), and — uniquely among engine tools — agentic-trace, which replays pre-built multi-turn agentic traces (e.g., OpenHands/SWE-smith) round by round, feeding the server’s real assistant reply back into the next turn’s history ². Arrival control mirrors vLLM: --request-rate with Poisson arrivals, inf for bursts, plus --max-concurrency. It benchmarks SGLang, vLLM, LMDeploy, TensorRT-LLM, and Truss endpoints through OpenAI-compatible or native APIs, reporting request/input/output token throughput, TTFT, ITL, and per-request latency.

genai-bench (SGLang project, pip-installable) is a standalone token-level benchmark with a live UI, traffic-scenario strings like I(1024,1024), concurrency sweeps, and Excel/plot report generation; it targets OpenAI-compatible and hosted-cloud endpoints and covers text, image, embedding, and rerank tasks.

GuideLLM (vLLM project) is the one the vLLM docs now recommend over vllm bench serve for production work: real and synthetic datasets (text/image/audio/video), execution profiles spanning Synchronous, Concurrent, Throughput, Constant, Poisson, and Sweep, multi-turn conversation benchmarking for chat agents, full client- and server-side tool-calling support, and Mooncake trace replay, with console/JSON/CSV/HTML reports.

llmperf (Ray-project fork of IBM’s original) is the lightweight veteran: a load test and a correctness test against OpenAI-compatible APIs, with Gaussian input-length sampling (--mean-input-tokens, --stddev-input-tokens), concurrency control, and inter-token latency / generation throughput per request. Its README carries a caveat that applies to the whole category: “the results may not correlate with users’ workloads” — it is a stress tool, not a realism tool.

llama-bench (llama.cpp) covers the single-machine local case: synthetic prompt-processing (pp), text-generation (tg), and combined (pg) tests with configurable prompt/generation lengths and batch sizes, outputting CSV/JSON/JSONL/SQL.

MLPerf Inference’s LoadGen is the standards-grade option: a reusable, model-agnostic load generator implementing the Offline, Server, and SingleStream scenarios with strict fairness and reproducibility rules ¹⁷. The LLM benchmarks (llama2-70b on OpenOrca, gpt-j on CNN-DailyMail; llama3.1-405b and mixtral-8x7b appeared in v6.0 but were dropped from the v6.1 list) pair LoadGen traffic with real datasets rather than synthetic tokens — compliance-oriented rather than exploration-oriented.

One notable absence: NVIDIA GenAI-Perf, historically the standard load-testing tool for Triton/TensorRT-LLM endpoints, could not be included. Its public GitHub repository and docs pages returned 404 at fetch time (September 2026) — only a PyPI package of the same name (0.0.16) remains reachable — so any current capability claims about it would be unverifiable.

Theme 3 — Trace datasets and calibrated generators: realism as a service

When you need production-shaped traffic, four options exist, in increasing order of modeling sophistication.

Raw trace replay. BurstGPT publishes real Azure ChatGPT/GPT-4 traces with per-request timestamp, session ID, model, input/output token counts, and conversation-vs-API mode; the README’s usage guidance is to scale the average RPS to your setup and optionally model the trace’s distributions ⁹. Azure’s official AzurePublicDataset repo adds representative LLM inference traces (November 2023 and May 2024) and multimodal inference traces with input/output token counts. LMSYS-Chat-1M provides 1M real conversations for chat-specific distributions ¹⁰; ShareGPT (via the HF ShareGPT_Vicuna_unfiltered dataset, the default in vLLM/SGLang) is the de-facto replay corpus. Replaying raw traces is realistic but static: it reproduces one slice of one provider’s history.

Statistical generators calibrated to traces. ServeGen is the current reference: a Python framework that composes workloads per-client from empirically fitted CDFs of arrival rate, burstiness (CV), and input/output length, with rate functions over time — covering language clients (m-small/m-mid/m-large), reasoning models (DeepSeek-R1’s bimodal thinking-length distribution), multimodal image workloads, hashed multi-turn conversation data, and anonymized offline batch traces ²². Its validation claim: workloads generated from its models produce more accurate performance benchmarking than naive generation. FineServe’s companion generator does the same for multi-model marketplaces, composing model-aware request mixtures with configurable shares of task intents ²⁵. BurstGPT’s paper similarly documents fitted burstiness models.

Engine-side trace ingestion. vLLM’s bench serve now ingests BurstGPT CSV directly as a dataset, and SGLang replays multi-turn agentic traces — the pragmatic middle ground where research-grade traces meet engine tooling.

The tradeoff, stated plainly. Raw traces are the ground truth but go stale (BurstGPT is GPT-3.5/4-era; reasoning-heavy traffic arrived later); fitted generators age better but are only as good as their source population — all three major trace families come from a single provider each (Azure, Alibaba, one marketplace), and ServeGen’s authors explicitly leave plugin/agentic calls to future work ²²²⁵.

Theme 4 — Agentic workloads: the new frontier

Agentic traffic breaks the assumptions the chat tooling was built on. The most complete characterization to date (ReAct-style agents on five benchmarks, reasoning and non-reasoning Gemma/Qwen models) finds agentic serving is stateful, multi-turn, and context-growing: each step consumes accumulated context, generates a mix of thinking, message, and tool-call tokens, and appends tool results ²⁴. Context growth is not a proxy for turn count — SWE-bench Pro averages 69K–80K accumulated tokens despite fewer turns than the turn-count tail workloads — and reasoning models can reduce total turns (Gemma Thinking averaged 18.0 turns vs 108.8 for Gemma Instant). The serving implication is structural: agent performance depends on preserving long-lived KV cache across turns; evict it and a decode-dominated workload becomes expensive recomputation, and concurrent long-context agents can exceed GPU memory ²⁴.

The benchmarking tooling is converging on trace replay with execution-graph semantics. XPerf, the most complete research framework found, traces real agentic applications (eight by default: coding agents like mini-SWE-agent, deep research like Open Deep Research/DeerFlow, Q&A), reconstructs each run’s execution graph (the DAG of LLM calls, tool calls, and dependencies), synthesizes new workloads from collected traces, and replays them deterministically — including enforcing output token IDs so that MoE routing and attention sparsity reproduce — while capturing serving metrics (KV-cache usage, prefix-cache hit rate, token throughput) and GPU metrics ²³. Its empirical results demonstrate why agentic benchmarking differs: mini-SWE-agent’s throughput degrades 35% as concurrency rises from 10 to 50 (prefix-cache thrashing), while DeerFlow’s throughput rises with concurrency (bigger decode batches, no cache to thrash) ²³.

Production-grade options are thinner. SGLang’s agentic-trace dataset mode is the first engine-native agentic replay ². GuideLLM supports multi-turn chat-agent benchmarking and full tool calling ¹. SemiAnalysis’ open-source InferenceX platform runs continuous benchmarks of major serving frameworks and models, and its AgentX benchmark is a long-context, multi-turn agentic workload (1M+ tokens) — useful as a public reference point even though it is not a load generator you run locally. The agent-task benchmark suites — τ-bench/tau-bench ⁷, AgentBench ⁸, TheAgentCompany ¹⁹, WebArena ²⁰, ToolLLM ²¹, GAIA ¹⁸ — generate the content of agentic traffic (task sets with tool environments) and can be adapted as workload sources, but they measure agent quality, not serving performance, and none of them models arrival processes. Note also that the tau-bench repo itself now warns its tasks are outdated and points to τ³-bench (tau2-bench) for current tasks.

Theme 5 — Simulating workloads before touching hardware

For design-space exploration — scheduler policies, parallelism, memory tiers — simulators replace the GPU with a calibrated model and generate workloads in software. LLMServingSim is the most mature: an iteration-level HW/SW co-simulator pairing a Python frontend that mirrors vLLM’s continuous-batching scheduler with the ASTRA-sim analytical backend, driven by per-hardware latency profiles; arrivals are synthesized (Poisson sampled from ShareGPT in the original paper) and it reproduces vLLM throughput trends closely ¹⁵. Version 2.0 extends to heterogeneous accelerators, disaggregated memory (CPU/CXL/PIM), MoE routing, and TP/PP/EP/DP combinations, and the project reports byte-identical results with an ~11x speedup and RTX 4090 profiles within 1% of real vLLM TTFT/TPOT ¹⁶. Frontier is a discrete-event simulator explicitly built for modern stateful workloads — reasoning agents, tool calls, and prefix-cache continuity across rounds. SimLLM takes the opposite fidelity split: the real vLLM/SGLang scheduler runs unmodified, while only the GPU executor and a packet-level network are simulated, yielding TTFT/TPOT/goodput on a virtual clock — useful when the question is about scheduling and networking rather than kernels. Simthesizer adds an agent-driven loop to explore the design space automatically ²⁶.

Theme 6 — Generating the content, not just the traffic

A distinct literature generates the text of workloads: prompts, tasks, and conversations. Self-Instruct bootstrapped the pattern (a seed set expanded by the model itself) ¹¹; Evol-Instruct (WizardLM) scales instruction complexity via iterative LLM rewriting ¹²; AgentInstruct generalizes it to multi-agent generative flows ¹³; distilabel is the maintained engineering framework for such pipelines; and RAGAS generates synthetic test data for RAG pipelines specifically ¹⁴. These matter for benchmarking because prompt content drives token-length distributions and cache behavior — but they generate datasets, not arrivals, so they compose with the load generators above rather than replacing them.

Where the evidence disagrees

Poisson vs. bursty arrivals. LoadGen’s scenario model, LLMServingSim’s ShareGPT-Poisson synthesis, and the --request-rate Poisson defaults in vLLM/SGLang all assume or default to Poisson arrivals ¹⁷¹⁵. BurstGPT, ServeGen, and FineServe all find real arrivals are bursty and non-stationary, and ServeGen’s KS tests show no single distribution fits across clients — including Gamma and Weibull ⁹²²²⁵. The best reconciliation: Poisson is a defensible baseline for interactive chat but a poor default for API-style workloads and any burst-sensitive experiment; the tools have now built the escape hatch (vLLM’s --burstiness, ServeGen’s per-client CVs), and the disagreement is really about whether the escape hatch is the default.

Determinism vs. representativeness. XPerf deliberately enforces output-token IDs so agentic runs are reproducible, accepting that this pins the execution graph ²³. The opposing view — embodied in raw trace replay and in the “results may not correlate with users’ workloads” caveat in llmperf — is that pinned or synthetic workloads lose the nondeterminism that makes agentic traffic what it is ²⁴. These are different measurements, not competing truths: XPerf answers “how does the system behave on this exact traffic,” traces answer “how did it behave on that day.”

What “realistic” means. ServeGen argues realism comes from fitting per-client statistical models of production traffic ²²; FineServe argues it comes from model-aware market mixtures ²⁵; BurstGPT argues the raw trace is the ground truth ⁹. The tension is population vs. mechanism: calibrated generators generalize better across settings but inherit one provider’s population; traces are authentic but frozen in time.

Gaps and open questions

• No standard agentic load generator. As of September 2026 there is no open tool that synthesizes agentic traffic from distributions the way ServeGen does for chat/API traffic — XPerf is the most complete framework but its repo was not findable at fetch time, ServeGen’s authors explicitly defer agentic/plugin calls, and SGLang’s agentic-trace replays collected traces rather than generating them ²³²²². The field needs a generator that models thinking-token ratios, tool-call distributions, and context growth across concurrent agents.
• Reasoning-model workloads. Thinking tokens change output-length distributions (bimodal, long) and cache economics; only ServeGen (DeepSeek-R1 mode) and InferenceX’s AgentX systematically address them, and no serving benchmark yet standardizes reasoning traffic.
• Multi-model market mixtures. FineServe shows market-driven model churn shifts demand sharply; no load generator yet composes heterogeneous model portfolios with realistic cross-model arrival coupling ²⁵.
• Trace staleness. The three public trace families (Azure, Alibaba, marketplace) cover 2023–2025 traffic and pre-date much of the agentic/reasoning era; fresh traces with agentic and reasoning labels are the highest-value missing artifact.
• GenAI-Perf’s disappearance. NVIDIA’s standard tool was unverifiable at fetch time (repo and docs 404; PyPI package still listed) — worth confirming before relying on any 2024-era GenAI-Perf material.
• Verification gap. None of the reviewed generators publishes cross-tool validation on the same hardware; claims like ServeGen’s “more accurate than naive generation” are validated only against their own traces ²².

Confidence and limitations

Confidence is moderate. The paper layer is solid: every citation was retrieved and DOI-verified this session, and the five load-bearing papers (ServeGen, XPerf, the agentic characterization, FineServe, LLMServingSim) were read in full text. The tool layer is vendor-documented: capabilities are quoted from READMEs and official docs fetched in-session and flagged as such, but were not executed on hardware, and several projects change weekly — vLLM and SGLang both restructured their benchmark tooling within the last year (scripts → CLIs), and TGI is now in maintenance mode. Limitations: GitHub search only surfaces repos by name/description keywords, so some tools will have been missed; docs for NVIDIA GenAI-Perf and several older tools were unreachable; commercial/hosted generators (e.g., AA-AgentPerf, referenced in XPerf) were out of scope; and the “agentic” category moves fast enough that some 2026 findings may be superseded within months.