Benchmarking tokens per watt: how AI inference energy efficiency is measured

Why this question

Tokens per watt has become the efficiency currency of AI infrastructure: accelerator purchases, serving-stack choices, power-capping decisions, and data-centre designs are argued in these units. But a unit is not a measurement. What the metric denotes (one GPU or one node or one facility; decode-only or end-to-end; one request length or another), how it is taken (a calibrated wattmeter, a software power model, or vendor telemetry), and on which workload it is taken all change the number by orders of magnitude. An architect comparing H100, B200, and TPU efficiency figures, or deciding whether power capping pays for a decode-heavy workload, is comparing numbers produced by different metrologies — and the published benchmarks are the only place those metrologies are described well enough to be compared.

What turns on the answer is expensive. The prior review in this series (model-to-grid AI datacenter efficiency) established what the efficiency levers are and how strong the evidence for each is. This review asks the measurement question underneath it: when a number like “tokens per watt” is reported — in a paper, a submission, or a vendor slide — what was actually measured, and can it be compared with any other number?

Scope and methods

Question. What benchmarks, metrics, and measurement methodologies exist for quantifying AI inference energy efficiency in tokens per watt (or per joule), from accelerator to data-centre level, and what do they establish about how inference efficiency should be measured and compared?

Inclusion criteria. Works published or posted 2019–2026 (the modern inference-efficiency literature), plus two metric-antecedent sources from 2009 and 2018 (SPECpower, Green500) because the modern metric descends from them; English; peer-reviewed papers and arXiv preprints (this fast-moving field lives partly in preprints); benchmark descriptions, measurement studies, energy- and carbon-per-inference quantifications, and metric or methodology critiques. Exclusion criteria. Training-only efficiency studies except where they define the metric landscape (Zeus, Patterson, Strubell are included for the measurement lineage, not as tokens/watt evidence); power-grid engineering; efficiency levers (power capping, scheduling, serving configuration) as such — already covered by the model-to-grid review — except where they report tokens/watt measurements; microarchitecture-only work; works unreachable through any open archive (see below).

Databases and dates. OpenAlex, Crossref, the arXiv API, arXiv abs pages, Semantic Scholar (best-effort; persistent 429s), Unpaywall for OA checks, the ACL Anthology for abstract rescue, and Zenodo/GitHub for a vendor-benchmark probe. Forty-eight query strings (listed in frontmatter) plus a 13-title landmark-resolution ladder (Crossref query.bibliographic → arXiv ti:) and citation-graph follow-ups. Searches run 2026-08-17.

Screening counts. 1,948 raw records retrieved (1,142 from OpenAlex, 806 from arXiv/Crossref); ~1,566 after DOI/title deduplication; 210 passed an automated topic-and-venue screen; 13 landmark titles were resolved through the resolution ladder (12 resolved cleanly, one resolved to a different but real paper that was kept on its own merits); 66 records were curated into the final selection. One source was dropped at retrieval — the Joule commentary “The growing energy footprint of artificial intelligence” (de Vries) — because every legitimate channel (cell.com, ScienceDirect, the institutional repository, OpenAlex, Crossref, Semantic Scholar) is bot-walled or abstract-less in-session; per this review’s protocol, unfetchable sources are not cited. Two SSRN preprints were dropped at screening (a weight-only-quantization energy study and “Green My LLM”) because SSRN blocks scripted access. A probe for the vendor benchmark “AI Energy Star” found no paper, DOI, Zenodo record, or GitHub repository — it exists only as marketing material, which is not citable under this review’s rules; the absence is noted in Gaps. 65 sources included, all retrieved this session: 20 read in full text, 45 at abstract level (flagged abstract-only and hedged accordingly). Every DOI was verified against Crossref (title similarity 1.00 on all checks; no retraction notices) or, for arXiv DOIs, by abs-page resolution; every 10.48550/arxiv.* DOI was confirmed to resolve.

What this review deliberately does not cover. Training-energy benchmarking as a topic (covered elsewhere in this series); the efficiency-lever evidence base (power capping, scheduling — see the model-to-grid review); vendor marketing figures with no citable artifact.

The landscape

The literature has three neighbourhoods that barely cite each other. The consortium-benchmark neighbourhood (MLPerf family, SPEC, Green500) defines what “efficiency benchmark” means institutionally: rules, submissions, comparability, and rankings in FLOPS/watt or queries-per-watt. The research-measurement neighbourhood (2023–2026) is where tokens-per-watt-style numbers actually live: wattmeter and telemetry studies of LLM inference on specific GPUs, serving engines, and cloud APIs. The carbon-accounting neighbourhood (FAccT, lifecycle-analysis venues) measures the same physical quantity but reports it in grams of CO₂ per query or per day, folding in grid carbon intensity and embodied manufacturing.

A vocabulary finding shapes everything below: the exact phrase “tokens per watt” appears rarely in the peer-reviewed literature. The field speaks in joules per token ⁸²⁶, watt-hours per request ²⁹, kWh per 1,000 inferences ⁴¹, and tokens per watt in the few papers that use the unit directly ⁶⁴. These are different quantities — a per-GPU decode-phase figure is not comparable with an end-to-end per-request figure — and converting between them requires request lengths, batch sizes, and measurement boundaries that most papers report incompletely. The unit heterogeneity is itself a comparability barrier, and no benchmark in the retrieved set standardises the conversion.

Theme 1: The metric lineage — from FLOPS per watt to energy per token

The efficiency metric did not start with language models. Green500 institutionalised FLOPS/watt rankings for supercomputers, with measurement-qualification rules that labs must satisfy to be listed ⁶³. SPEC established SPECpower_ssj2008 as the first industry-standard server power-and-performance benchmark ⁶. The “Green AI” agenda then made efficiency a first-class evaluation criterion for machine learning, documenting an estimated 300,000× growth in deep-learning compute from 2012 to 2018 ⁶⁵. Strubell, Ganesh and McCallum supplied the first systematic energy accounting for NLP — including inference, not just training — and the policy recommendations that followed ⁴⁴.

The MLPerf family carries this lineage into machine learning. The training suite was characterised in a peer-reviewed study of its workloads versus earlier benchmarks such as DAWNBench and DeepBench ²; the inference benchmark spans three orders of magnitude in system power and five in performance, and its closed division enforces latency constraints precisely because throughput-only comparisons mislead ³. The mobile and tiny variants extended the family down the power axis — MLPerf Tiny made energy a first-class reported metric alongside accuracy and latency ⁴⁵.

Against this background, the token-based unit crystallised in 2024–2026. A workshop paper argued that Energy-per-Token should complement accuracy benchmarks, showing that test-time-compute strategies can trade enormous energy for accuracy (chain-of-thought on MATH raised accuracy by 281% at a reported +15,132% energy cost) and that smaller models with controlled reasoning beat larger models on an energy-accuracy frontier (a 40–60% energy saving at similar or better accuracy) ²⁶. TokenPowerBench formalised joules-per-token as its headline metric ⁸, and the 1/W law paper defined tokens/watt explicitly, decomposing it into single-GPU and fleet-level quantities ⁶⁴. A survey of green deep learning catalogued the same landscape from the training side and flagged the recurring gap between FLOPs-based and runtime-based efficiency claims ⁴⁹, and an analysis of inference-energy trends argued that inference energy follows different laws than the performance-versus-parameter scaling curves ²⁷.

Theme 2: The benchmarks that measure inference efficiency

MLPerf Power is the closest thing to a standard. Developed by a consortium of more than 20 organisations, it establishes rules and best practices for measuring ML-system energy efficiency from microwatts to megawatts, and reports 1,841 reproducible measurements across 60 systems ¹. Its significance is institutional: it is the only benchmark in this review whose numbers are produced under agreed rules rather than per-lab conventions. Its limitation is scope: its workloads are the classic MLPerf inference set, not generative-LLM serving, so tokens/watt for chat workloads is outside its remit.

The MLPerf-based evaluations that do exist are instructive about comparability. One study measured four accelerators (NVIDIA A100, Intel Gaudi, Graphcore Bow-Pod64, Groq LPU) on two MLPerf workloads to a common target accuracy: which accelerator is most efficient depends on the workload — Gaudi2 had the lowest energy on ResNet50, the A100 on BERT-Large inference ¹¹. A 2026 study of 739 MLPerf Inference v5.0 submissions found no significant public-versus-private-cloud efficiency difference but large accelerator-model differences (the H200 showing roughly 45% higher median throughput-per-TDP efficiency), concluding efficiency scales multiplicatively across accelerator generations ¹⁰. Both results cut against any single “best accelerator” narrative.

The research benchmarks report the tokens-per-watt-style numbers. TokenPowerBench measures GPU-, node-, and system-level power without specialised meters, attributes energy to prefill and decode phases per request, and reports joules per token across four model families from 1B to 405B parameters, with batch size, context length, parallelism, and quantisation as configurable dimensions ⁸. GPU-NEST established an energy-efficiency characterisation methodology for multi-GPU inference servers and showed power as a first-order constraint in them ⁷, and WattWiser targeted power- and resource-efficient scheduling for multi-model multi-GPU inference servers ²⁵. From Words to Watts benchmarked LLaMA models on V100 and A100 clusters: a 65B model consumed roughly 3–4 joules per output token at 512-token generation, and power-capping the GPUs from 250 W to 175 W cut total energy by 23.21% at a 6.7% time penalty ¹⁸. How Hungry is AI benchmarked 30 models in commercial data centres through their APIs, combining public performance data with environmental multipliers: the most energy-intensive models exceeded 29 Wh per long prompt — over 65× the most efficient — and a 0.42 Wh short query, scaled to 700 million queries per day, aggregates to annual electricity comparable to 35,000 US homes ³¹. A NAACL 2025 study benchmarked inference energy across NLP tasks and found energy correlates strongly with output-token length and response time, with quantisation, optimal batch sizes, and prompt phrasing as significant levers ³⁰. An IEEE Access 2024 profiling study reached a similar conclusion from the architecture side: model size, layer count, parallelised attention, and vocabulary size drive inference energy, while batch size and quantisation level are the tunable levers ¹⁹. A cluster-redesign study for GPT-Neo reported substantial throughput, latency, and energy improvements from a novel interconnect and architecture, though without retrievable magnitudes ²⁰.

A second strand benchmarks the serving engines rather than the models. One study decomposed LLM inference engines (vLLM, TensorRT-LLM, DeepSpeed) on a two-H100 node into setup and token-generation stages to locate energy bottlenecks ⁹. Energy-cost models extend this: a workload-based model fitted energy and runtime across heterogeneous CPU-GPU systems with R² above 0.96 ²², and a token-count-aware allocation strategy across heterogeneous clusters saved 7.5% of energy versus a workload-unaware baseline ⁵⁹. On the edge, 28 quantised LLMs were measured on a Raspberry Pi 4 ³², an energy-aware DVFS-plus-speculative-decoding scheme reported a 52.4% energy reduction on-device ²⁴, and measurements on Apple Silicon found energy per token grows monotonically but weakly sublinearly with prompt length ¹³ — with Apple Silicon reported at 4–7× less energy per request than NVIDIA-based servers on interactive workloads in one comparison ⁶¹. Model-level energy estimates round out the set: aggressive weight-only quantisation cut Llama-2-7B inference energy by 4.6× and moved the bottleneck to attention ²³, and pre-storing attention matrices cut energy 1.45–2.83× against baselines ⁶².

Theme 3: The metrology — how the numbers are taken

Underneath the benchmarks sits a measurement-methodology literature, and it is where the comparability problems live. Physical power meters are accurate but expensive and cannot attribute energy to individual services; software-based power meters (power models plus vendor interfaces) are deployable at scale, and a 2023 study compared the leading tools for CPU and GPU measurement, concluding the choice of tool materially changes what you can claim ¹⁴. A review of energy-estimation approaches catalogued the same trade-off for machine learning generally ⁵², and a 2025 MLOps study applied software-based power measurement across discriminative and generative pipelines, concluding architecture and hardware choices dominate pipeline energy ⁵³. An energy-consumption-index proposal aims to standardise cross-architecture DL comparison, though it has been validated on CNNs rather than LLMs ⁵⁴. On the attribution side, WattScope estimates application-level power from a server’s aggregate draw without OS access, exploiting the low-variability, periodic power signatures of data-centre workloads ¹⁶, and a EuroSys 2026 paper tackles the harder version — attributing GPU power to individual jobs in shared cloud settings ¹⁵.

Phase attribution is the emerging standard for LLM-specific measurement. TokenPowerBench aligns metrics to prefill versus decode per request ⁸; the engine study separates setup from token generation ⁹; and the CCGrid 2026 characterisation shows why phase matters — decode-phase frequency scaling from 2842 MHz to 180 MHz saves 42% of energy at only 1–6% latency cost, while prefill behaves differently ¹². Scale matters for methodology too: the BUTTER-E dataset, 63,527 watt-metered runs, found GPU training used a median 9.47 mJ per datum versus 6.16 mJ on CPU in its setting, and that a fitted energy model predicts GPU energy within ±2.74% but CPU energy only within ±20.2% ¹⁷.

Reporting practice lags all of this. A 2020 audit found that only about 1 in 100 sampled NeurIPS papers reported energy at all, and proposed a systematic reporting standard; the audit’s own experiments consumed 24.344 kWh and 8.021 kg CO₂eq ⁴⁸. The configuration space is the final confound: power capping is simultaneously an optimisation lever and a measurement choice. The same GPU measured at different caps produces different tokens-per-watt — 250 W versus 175 W changed total energy by 23.21% on one workload ¹⁸, and running at 1.6 GHz instead of 2.0 GHz held throughput constant at under 80% of the energy ²⁸ — so an efficiency number without its power configuration is almost meaningless. Idle power compounds this: the 1/W law analysis shows the B200’s 430 W idle draw becomes a dominant fraction of the bill at long context, where KV-cache limits cut concurrency ⁶⁴.

Theme 4: What the measurements show

The verified numbers cluster into four findings.

First, per-token energy at the server level is in the single-digit joules on current data-centre GPUs, with a wide spread. A 65B LLaMA on A100s: roughly 3–4 J per output token ¹⁸. Llama-2-70B serving: 0.77 to 13.21 Wh per request depending on request class, tensor parallelism, and frequency ²⁹. Per query, 2.9 J for a 1B model rising to 21.0 J for a 32B model — a 7.2× model-size spread ¹².

Second, the task and workload spread dwarfs the hardware spread. Per 1,000 inferences, energy ranges from 0.002 kWh for text classification to 2.9 kWh for image generation — a factor above 1,450 — with multi-purpose generative models orders of magnitude costlier than task-specific ones ⁴¹. In commercial data centres, the model spread is 65× on long prompts ³¹. Anyone comparing “tokens per watt” across studies is comparing numbers that vary more with workload than with hardware generation.

Third, inference, not training, dominates steady-state energy — which is the entire justification for the metric. Industry reports cited by TokenPowerBench put inference above 90% of total power consumption ⁸; a comparison study estimated GPT-4’s training at ~9,450 MWh versus inference exceeding 500 MWh per day ⁵¹; BLOOM’s API deployment was estimated at ~19 kg CO₂eq per day versus 24.7–50.5 tCO₂eq for training ⁴²; and the serving phase has now surpassed training in energy terms per one 2024 assessment ⁵⁸.

Third-and-a-half, model size and accuracy do not move together on energy. Across 14 open-source LLMs (6B–34B) on an A100, larger models often consumed substantially more energy without proportional accuracy gains, with mid-sized models matching larger ones at much lower energy ³³. Prompt-engineering choices also change the number: prompt phrasing measurably altered Llama 3’s inference energy on code generation ⁵⁵.

Fourth, the measured levers are large but regime-dependent. Power/frequency capping: 23.21% energy saved at 6.7% time cost ¹⁸, ~20% power reduction with zero latency impact ²⁸, 42% via decode-phase DVFS ¹², and near-zero impact at a supercomputing centre where jobs were not power-bound ³⁶. Routing small queries to small models: 80% energy saving at a 6.8-point quality cost, 88% combined with DVFS ¹². Cluster-level design: 53% energy, 38% operational carbon, and 61% customer-cost reduction under latency SLOs in DynamoLLM’s evaluated design ²⁹.

A related but distinct literature reports carbon per query, which is the same measurement problem with a grid-intensity multiplier: LLMCarbon models end-to-end carbon within ≤8.2% of measured footprints ⁴⁶; EcoServe’s carbon-aware design cut emissions by up to 47% with embodied carbon exceeding 50% of lifetime in its analysis ⁵⁶; Clover reduced inference-service carbon under SLA constraints ⁵⁷; and cloud-side estimates argued that provider-reported software carbon intensity is the missing prerequisite for mitigation ⁴⁷. Patterson et al. quantified the upstream lever space — efficiency and siting together can reduce training carbon by 100–1,000× ⁴³ — and a longitudinal analysis found embodied carbon roughly equal to operational footprint and ~20% operational-power reduction per six months of hardware improvement, at an assumed PUE of 1.1 ⁴⁵. A workload model of ChatGPT-class inference compared request-routing policies — local, balanced, and carbon-minimising — for their power and carbon impacts ³⁴, and a life-cycle analysis of LLM-powered chatbots identified eight energy- and carbon-relevant phases with three strategic mitigation pathways ⁵⁰. At the serving-system level, a cloud-scale characterisation of ML serving reported up to 28.3% energy savings from hierarchical GPU power management across 105 servers ²¹.

Theme 5: The facility-level gap — tokens per watt for data centres

The review’s central negative finding: no retrieved benchmark or measurement study reports tokens per watt at the data-centre level. The facility literature measures efficiency in PUE — the ratio of total facility energy to IT energy. Studies tune PUE with sensor-plus-ML methodology ³⁹, model PUE statistically across locations ³⁸, measure a small data centre end-to-end including cooling and demand response ⁴⁰, and — most tellingly — an integrative review asks whether PUE is still fit for purpose for AI infrastructure at all ³⁷. Production-scale power trends are tracked at the facility level (NERSC’s machines stayed well below provisioned power even after GPU transitions ³⁵), and supercomputing centres manage power draw through capping ³⁶.

The bridge between tokens and facilities exists only in fragments. MLPerf Power spans microwatts to megawatts but measures systems, not facilities ¹. One 2026 preprint treats tokens served as the dispatchable unit for data-centre demand response, cutting operating cost 34.3% without curtailing token volume ⁶⁰. And the facility metric that would complete the chain — tokens per watt at the wall, i.e. server-level tokens/watt divided by facility overheads — appears nowhere as a measured quantity. Every published number in this review stops at the server or node boundary; the data-centre boundary is reached only by arithmetic.

Where the evidence disagrees

What the metric should optimise. The energy-per-token advocates argue efficiency belongs alongside accuracy in benchmark scores ²⁶; the MLPerf tradition treats efficiency as one dimension of a comparability regime built for performance ¹; the carbon literature argues neither energy nor performance is the right objective — grid intensity and embodied carbon are ⁵⁶⁵⁷⁴⁷. These are disagreements about objectives, not numbers, but they change which benchmark you build.

Does quantisation save energy? Yes, measured 4.6× on Llama-2-7B with weight-only quantisation ²³, and quantised edge models are the default for efficiency ³². But the one peer-reviewed-adjacent counter-evidence — an empirical study arguing weight-only quantisation does not always save energy across NVIDIA platforms — exists only as an SSRN preprint that was unreachable in-session and is therefore not cited here. The tension is real in the field and unresolved in this review.

Which engine or accelerator wins? Rankings do not transfer across workloads: Gaudi2 beats A100 on ResNet50 energy but loses on BERT inference ¹¹; engine efficiency depends on whether you measure setup or generation phase ⁹; and the accelerator-generation effect dominates any cloud-type effect in the MLPerf submission record ¹⁰. Any single ranking is an artefact of its workload.

Power capping: free lunch or nothing? Large savings on some workloads ¹⁸²⁸, 42% on decode with 1–6% latency cost ¹², and minimal impact at a centre whose jobs were not power-bound ³⁶. The disagreement tracks workload memory-boundedness — the same regime dependence the model-to-grid review found — and reinforces that tokens-per-watt numbers are configuration-relative.

Energy versus carbon. Optimising energy is not optimising carbon: the grid’s marginal intensity, not the GPU’s watt draw, decides emissions ^56574347. Studies that report both (DynamoLLM’s 53% energy versus 38% carbon savings ²⁹) show the gap explicitly.

Gaps and open questions

• No facility-level tokens/watt benchmark exists. Nothing in the retrieved literature measures tokens per watt at the data-centre boundary. Settling this requires a consortium-standard benchmark that measures LLM serving at node and facility level with a fixed boundary, workload mix, and phase attribution — the MLPerf Power model extended to generative serving and to the wall.
• No consortium benchmark covers generative-LLM serving energy. MLPerf Power’s workloads predate it; TokenPowerBench is research-grade; the commercial-DC numbers come from API-side estimation, not independent measurement ³¹. The field is benchmarking around the edges of the workload that matters.
• No independent cross-cloud measurement. The largest comparative dataset (739 submissions) is vendor-submitted ¹⁰; the cloud-carbon literature argues providers must publish software carbon intensity for anyone to act ⁴⁷.
• Measurement boundaries are unstandardised. GPU-only versus node versus system; idle power included or excluded; phase attribution present or absent; power configuration reported or not. Henderson’s proposed reporting standard ⁴⁸ has not been adopted by any major benchmark.
• Vendor benchmarks are uncitable. The “AI Energy Star”-style figures circulate in marketing with no paper, DOI, or repository; this review can neither verify nor cite them, and the ecosystem’s reliance on them is itself a gap.
• Idle power and long-context behaviour are under-measured. The 1/W law ⁶⁴ suggests the efficiency cliff at long context is physical (KV-cache concurrency), but it is one preprint’s analytical model — H100-calibrated, B200-scaled — awaiting independent measurement.

Confidence and limitations

Confidence is moderate. The 20 full-text sources carry the quantitative claims in Themes 2 and 4, and every load-bearing number was re-verified against the retrieved full text or abstract before citation. The 45 abstract-only sources (mostly paywalled IEEE and ACM items) are cited for qualitative claims only. Specific limitations: (1) the de Vries Joule estimate — probably the most-cited per-request energy figure in public discourse — could not be retrieved through any legitimate channel and is not cited; (2) two SSRN preprints (weight-only-quantisation energy study, “Green My LLM”) are excluded for the same reason, leaving a known counter-voice out of the quantisation disagreement; (3) the “AI Energy Star” probe found no citable artifact, which is evidence about the vendor-benchmark ecosystem but not proof of absence; (4) English-only sources, 2009–2026 with the two metric-antecedent exceptions noted; (5) the facility-level gap is an absence finding — no retrieved source measures it, which is strong evidence of a gap but not proof none exists.