The LLM compute market — training, fine-tuning, and inference

Why this question

Capital allocation in AI infrastructure is being decided on a split nobody can directly observe. Tens of billions of dollars of GPU capacity are contracted on the assumption that inference demand will eventually dwarf training demand, yet the public accounting of that split is fragmentary: analyst houses measure different flows, vendors report different mixes, and fine-tuning — the middle category — has almost no dedicated market sizing at all. For an AI factory builder, the question is not academic. A factory’s architecture, power contract, network topology, and customer base all depend on whether its capacity will be consumed by six-month training runs or by continuous, latency-sensitive, bursty inference traffic. And the strategic question — whether to move downstream into inference — is a channel-conflict decision: the company’s customers are inference providers, so the answer determines whether the factory builder is a neutral supplier or a competitor.

Scope and methods

The question has two halves: a market-data half (size, split, growth of LLM compute demand by activity) and a strategy half (what the shift means for an AI factory builder). Because the evidence for the first half is almost entirely industry-produced — analyst reports, vendor financials, and consulting surveys rather than peer-reviewed literature — this review uses a two-layer evidence model. Peer-reviewed and preprint sources (28, all with verified DOIs, fetched and read this session, mostly abstract-level) anchor the mechanisms: scaling laws, serving systems, disaggregation, fine-tuning methods, energy accounting. Industry reports and primary vendor documents (Menlo Ventures, Bloomberg Intelligence, Gartner coverage, NVIDIA releases, SEC filings, Epoch AI, Stanford’s AI Index, pricing pages) are cited inline with URLs in the body; they carry no DOIs and so cannot be entries in the reference list, but every number attributed to them was verified by fetching the document this session. Screening: 543 raw records from 33 search queries across OpenAlex, Semantic Scholar, Crossref, and arXiv were deduplicated to 417, screened to ~30 academic candidates, and 28 included; 4 remembered landmark titles failed resolution and were dropped rather than forced. Industry-side: ~30 primary documents were fetched and grepped for the load-bearing numbers; figures that could not be verified (e.g. CoreWeave margin commentary, specific GPU-utilization percentages, an un-sourced “80% of compute is inference” claim) are excluded or flagged. A second search wave (2026-08-19) targeted the gaps flagged in the first pass: direct search engines (Google, Bing, DuckDuckGo, Mojeek, Startpage) bot-blocked this host, so the wave used Google News RSS, publisher WordPress REST APIs, and primary PDFs — it added the Stanford AI Index 2026 report ⁷ and two inline industry sources (NVIDIA’s token-factory technical blog and I/O Fund’s neocloud-financing analysis), and it confirmed several gaps remain (the Gartner “$112B agentic spend by 2028” figure, Deloitte’s agentic-spend share claim, VentureBeat’s GPU-utilization analysis, and coverage of CoreWeave’s 2026 price increase could not be retrieved to a verifiable primary). The review deliberately does not cover application-layer AI revenue, chip design details, or regional markets beyond what the sources report.

The landscape

The market-economics literature is thin relative to the stakes, and it is dominated by a few institutions. On the academic side, Epoch AI’s compute-trend work (training compute doubling every ~5 months, data exhaustion forecasts) and the Stanford AI Index are the canonical references, while the serving-systems literature (Splitwise, DistServe, vLLM, SGLang, DynamoLLM, MegaScale-Infer) is the most rigorous evidence on what inference actually costs to operate. On the industry side, the usable market numbers come from a handful of houses — Menlo Ventures (enterprise spend), Bloomberg Intelligence (long-horizon revenue), Gartner and IDC (IT spending), Epoch AI (compute stock and frontier-lab economics) — plus NVIDIA’s own disclosures, which are the single most influential primary source because NVIDIA sits at the revenue bottleneck of the whole chain. The landscape has two notable absences: nobody publishes a standalone fine-tuning market size, and nobody publishes a defensible split of training compute by modality. Both absences are themselves findings, discussed below.

Theme one — The market: total size and the training-to-inference rotation

The top-line numbers agree that AI spend is enormous and growing, but they measure different flows. Gartner forecasts $2.59T of total AI procurement in 2026 (up 47%), including $401B of AI infrastructure, while IDC counts $487B of AI infrastructure hardware in 2026 (up 53%) and a cumulative run past $1T by 2029 — the gap between the two is scope: procurement vs hardware shipments. Bloomberg Intelligence projects $1.8T of generative-AI revenue by 2032, up from ~$93B in 2023, and IDC’s deck sees enterprise AI platforms and services growing from $400B in 2026 to $2.1T by 2029. On the investment side, the Stanford AI Index 2026 reports US private AI investment of $285.9B in 2025 — more than 23 times China’s $12.4B — with global corporate AI investment more than doubling and private investment up 127.5% ⁷. These are not additive, and the review treats them as a range, not a consensus point.

The split that matters — training vs inference — is where the sources converge on direction and disagree on timing. Bloomberg Intelligence’s March 2025 deep dive puts the inference market at $735B by 2032 versus $580B for training, and expects inference compute consumption to overtake training “at least three years sooner than our initial projections” — by 2029 — driven by reasoning models and agent deployment. Gartner’s August 2026 release goes further: inside the AI-optimized infrastructure-as-a-service market, inference spend ($23.3B) already exceeds training spend ($19B) in 2026, rising from 55% to 59% of that market in 2027. Epoch AI’s numbers for the frontier labs run the other way — only ~30% of OpenAI’s compute spending in 2024 went to inference — which is exactly the expected pattern: the labs that train frontier models are the last place where training dominates. The 80/20 heuristic (inference as ~80% of long-run AI compute) circulates widely in investment commentary; this review treats it as directional, because the three measured datapoints above (BI 2032, Gartner 2026, Epoch/OpenAI 2024) tell a coherent story without it: inference crosses over first in the broad commercial market, later at the frontier labs themselves.

Two mechanisms explain the rotation. First, the training side is supply- and data-constrained: training compute has grown ~5x per year since 2020 ¹, but the Chinchilla scaling law says the efficient frontier pairs compute with data ², and public human text is forecast to be exhausted somewhere between 2026 and 2032 ⁴ — repeated data only stretches so far ⁵. The 2026 AI Index adds a concentration signal: fewer notable models were released in 2025 than the year before, parameter counts have plateaued near 1 trillion, and the frontier labs have stopped disclosing training compute ⁷ — training demand is narrowing to fewer, larger runs. Second, the inference side is demand-multiplied: reasoning models emit long chains of thought before answering ⁹, open-weight releases compress prices and expand the market ⁸, and Goldman Sachs (cited in the Gartner coverage) projects agent token consumption growing 24x between 2026 and 2030 to ~120 quadrillion tokens per month, even as per-token inference costs fall ~60% a year. NVIDIA’s own disclosures corroborate the demand side: inference token generation “surged tenfold in just one year” (Q1 FY26), demand is “compounding across training and inference — each growing exponentially,” and cloud GPUs are “sold out” (Q3 FY26). The uncertainty that runs through all long-horizon forecasts is captured well by ²⁸: explosive-growth arguments depend on strong assumptions about how fast automation spreads, and they are exactly the assumptions the market is betting on.

Theme two — Fine-tuning: a real but small slice

Fine-tuning is the least-measured category in the whole market. No major analyst house publishes a standalone fine-tuning spend figure; it is folded into “training” by Bloomberg Intelligence and into “model training infrastructure” by Menlo Ventures — a $4.0B line in 2025 that covers training and adaptation together. The adoption evidence suggests the slice is small: Menlo’s 2024 survey found only 9% of enterprise production models were fine-tuned, and its 2025 edition still describes fine-tuning as a niche technique relative to prompt design and RAG. The mechanism-level literature explains both why fine-tuning is cheap enough to exist as a category and why it stays niche. Parameter-efficient methods collapsed the cost: LoRA cuts trainable parameters ~10,000x and GPU memory ~3x versus full fine-tuning ¹⁸, and QLoRA fine-tunes a 65B model on a single 48GB GPU ¹⁹. But the quality evidence is mixed: in standard low-rank settings LoRA substantially underperforms full fine-tuning in-domain, even though it forgets less out-of-domain ²⁰ — a trade-off that pushes many production teams toward retrieval and prompting instead. The honest summary: fine-tuning is a real workload with real infrastructure demand (and the only category where “adaptation” compute is visibly growing with open-weight adoption), but every available proxy says it is single-digit percent of LLM compute spend, and the review found no source that contradicts that.

Theme three — Inside inference: chat, code, and the agentic wave

The best empirical map of inference workloads is the a16z/OpenRouter study of over 100 trillion real tokens, which finds that programming and creative roleplay are the two dominant token categories — roleplay alone is ~52% of open-source-model tokens — and that average prompts have grown ~4x since early 2024 (from ~1.5K to over 6K tokens) while completions nearly tripled, driven by reasoning tokens. Menlo’s enterprise data shows the professional side: coding is the first killer use case at ~$4B of 2025 spend (55% of departmental AI), and model-API providers (Anthropic, OpenAI, Google) capture 88% of enterprise LLM API usage — an inference-priced consumption line inside the $19B application layer.

The structural shift is agentic. The academic lineage is clear: ReAct established the reason-then-act loop ²³, Toolformer showed models can learn to call tools ²⁴, and the agent survey literature documents the resulting architectures ²⁵. Each agentic turn converts one user task into multiple model calls with tools, memory, and re-planning — which is why the OpenRouter study says “agentic inference will be taking over the majority of the inference,” and why Gartner describes an “Inference Paradox”: better unit economics encourage more powerful models, more tokens, and more complex workflows, so per-workflow inference costs rise more than 5x through 2028 even as per-token prices fall. The gap between aspiration and reality matters for capacity planning: Menlo finds only 16% of enterprise and 27% of startup deployments qualify as true agents today, and agent platforms are ~10% ($750M) of the $8.4B horizontal-AI category — so the agentic wave is a forecast backed by directionally strong evidence (token mix, reasoning-model adoption, tool-call growth) rather than an already-realized majority of spend.

Theme four — Training’s modality mix: multimodal-first, text-centric accounting

On modality, the honest finding is that the split of training compute between text-only and multimodal work is not publicly measured. No analyst house or academic source this review could find publishes it. What the evidence does establish: frontier models are natively multimodal — Gemini was introduced as a family spanning image, audio, video, and text understanding ²⁶ — and the modal frontier since 2024 has been omni-input (GPT-4o-class) plus video generation and world models. The AI Index documents the broader trend of model releases shifting toward multimodality (its 2024 report tracks technical advancement and industry statistics; the report series is the standard reference for such counts) ⁶. The one modality datapoint with real market grounding is the open-source download mix: text-generation models grew to 42% of downloads in 2025 while classifier, image, and audio models collapsed from ~70% in 2022 to under 6%, with multimodal and video-generation models growing in their place ⁷. But compute accounting remains text-centric: Epoch AI’s flagship training-compute series tracks frontier language models specifically, and the training-scaling literature that anchors the whole market discussion — ¹²⁴⁵ — is built on text-LM loss. Two indirect signals suggest multimodal’s compute share is rising from a small base: video-generation and world-model training is orders of magnitude more compute-hungry per training token than text (diffusion-style models at frontier scale), and the data-constraint argument that caps text pretraining does not bind on video, which is comparatively abundant. The gap matters: if multimodal training compute is 10% or 40% of the frontier, datacenter power planning changes materially — and nobody publishes the number.

Theme five — Utilization and the economics of the AI factory

Utilization is where the profit of an AI factory is actually made, and the evidence separates cleanly into training and inference regimes. Training utilization is a well-studied engineering problem: PaLM set the standard metric (model FLOPs utilization, 46.2% on TPUv4) ³, and MegaScale shows the ceiling at scale — 55.2% MFU on 12,288 GPUs, with fault tolerance and stragglers dominating operational effort ¹⁵. Inference is the harder utilization problem. Splitwise’s characterization is the anchor: generation is memory-bound and underutilizes compute even with state-of-the-art batching ¹⁰; DistServe shows that colocating prefill and decode creates interference that couples resource allocation and violates per-phase SLOs ¹¹; and MegaScale-Infer shows MoE inference specifically drops GPU utilization because sparsity makes the feed-forward layers memory-intensive ¹⁶. Serving software is the lever that closes part of the gap — PagedAttention’s KV-cache management ¹², SGLang’s 6.4x throughput via RadixAttention ¹³, and the broader efficiency taxonomy of quantization, pruning, and distillation ²⁷ — which is why inference cost per token keeps falling ~60% a year even as demand compounds.

Energy is the second-order constraint. The accounting tradition starts with Strubell’s training-energy estimates (up to 626,155 lbs CO2e for a neural-architecture-search run) ²¹, but the balance has flipped: recent work finds inference energy is now the dominant and growing LLM energy cost ²², and per-request energy scales with model size ¹⁷. The AI Index 2026 draws the same conclusion with harder numbers: it reports estimates that the energy required to serve inference queries can exceed the one-time cost of training within months, that annual GPT-4o inference water use alone may exceed the drinking-water needs of 1.2 million people, and that US AI data-center power capacity has risen to 29.6 GW ⁷. This is the layer where the prior reviews on this site go deep — tokens per watt as the measurement frame tokens-per-watt, the serving-engine tricks that move the number llm-inference-engine-tricks, and datacenter-level efficiency model-to-grid-ai-datacenter-efficiency. NVIDIA has now productized the framing: its technical blog markets the AI factory as a token factory, claims a 1,000,000x increase in inference throughput per megawatt across six GPU generations, and states the operating principle bluntly — “every watt not spent on cooling or idle capacity becomes a watt that generates tokens.” The market context from the other industry sources: Epoch AI tracks the AI-chip stock doubling roughly every 6.8 months and 83 covered US AI datacenters totaling 12.7 GW of IT power, Anthropic’s policy paper demands at least 50 GW of US capacity by 2028, and NVIDIA now markets “10x throughput per megawatt” as a benchmark win — tokens-per-watt has become a headline sales attribute of the factory itself.

The economics at the factory layer are brutal and public. CoreWeave’s FY2025 10-K is the clearest window: $5.1B revenue, a headline gross margin around 72%, but a separate $2.9B “technology and infrastructure” line (depreciation, power, operations) that leaves operating income at −$46M for the year; 98% of revenue came from committed contracts; Microsoft alone is 67% of revenue; OpenAI committed up to ~$6.5B through May 2031. The financing layer that keeps this running is now visible too: I/O Fund’s analysis of the two public neoclouds documents ~3.5 GW of contracted power capacity each (CoreWeave targeting 1.7 GW active by end-2026, Nebius 800 MW to 1 GW), Microsoft commitments of roughly $60B across neoclouds, and a circular-financing structure — NVIDIA equity, hyperscaler contracts, GPU-backed debt — that funds the buildout while the operators remain far from profitable. Sequoia’s “$600B Question” frames the same structure from the demand side: NVIDIA’s revenue run-rate, doubled for datacenter TCO and doubled again for end-user gross margin, implies ~$600B/year of end-user AI revenue is needed to justify the buildout — a gap Sequoia sized at $500B in mid-2024 and growing. NVIDIA’s own gross margin (~71% GAAP for FY2026 on $215.9B revenue) is the top of the pyramid; the factory operators are the squeezed middle; and the inference providers on top live on cents per million tokens (Together AI lists frontier-class output at $1.20–$15.00 per 1M tokens, with cached-input discounts; Lambda rents B200s at $9.86/GPU-hour falling to $8.87 at scale). The chain is thin in the middle — which is precisely where an AI factory builder operates.

Theme six — Strategy: should an AI factory builder move into inference?

The strategic question has a clear answer in the evidence: do not sell inference-as-a-service; do become inference-workload-aware as infrastructure.

The competitive case is structural, not just precautionary. An AI factory builder’s customers are inference providers, and those relationships are large, multi-year, and pre-committed — the CoreWeave pattern (98% committed revenue, OpenAI at ~$6.5B through 2031, Microsoft at 67% of revenue) is the industry’s dominant contracting model. Launching a token business would ask those same customers to fund a competitor for the thinnest-margin, most competitive layer in the chain, while destroying the one property that makes factory capacity valuable: fungibility. NVIDIA’s own articulation is the cleanest statement of the neutrality principle — “the factory can be used by another customer, another cloud or another operator” — and NVIDIA’s behavior is the template: it backs PORTS-Pike (4.25 GW initial, OpenAI as the tenant, ~$600B of compute through 2030 across OpenAI’s ~12–16 GW of commitments), it monetizes inference through silicon, systems, software (Dynamo, DGX Cloud, the DSX factory platform), and it never sells tokens — its technical blog’s “token factory” framing is explicitly about maximizing revenue per megawatt of infrastructure, not about selling tokens itself. The pure-play champion agrees: CoreWeave markets Mission Control for “provision infrastructure, schedule and manage workloads, and monitor performance across training and inference environments” — inference-awareness as infrastructure, no model-serving product line. Straddling exists (Together AI sells both token APIs and GPU clusters), but the straddlers serve developers, not competing inference providers; for a factory builder whose counterparties are inference companies, neutrality is the product.

The value-add case is that inference-workload-aware IaaS is a real, sellable product layer that does not cross the line. The systems literature defines the design space concretely: disaggregated prefill and decode with per-phase SLOs (TTFT/TPOT) ¹⁰¹¹, MoE-aware serving that lifts GPU utilization ¹⁶, cluster-level energy management under SLOs (DynamoLLM conserves 53% energy and cuts customer cost 61% while meeting latency targets) ¹⁴, and the serving software that compounds efficiency ¹²¹³²⁷. Each of these changes factory architecture — separate prefill and decode pools, KV-cache-aware fabrics, power capping per workload class, burst absorption for agentic traffic, co-locating training and inference to smooth utilization — and each is an infrastructure attribute that can be engineered, guaranteed, and priced without the builder ever serving a token. The market is already monetizing fragments of it: CoreWeave sells scheduling and monitoring across training and inference; I/O Fund’s analysis of the neoclouds names “optimized compute utilization” as one of the three reasons hyperscalers commit billions to them — utilization-aware operation is the actual selling point of the pure plays; Together sells “token-based capacity with SLAs” at the serving layer; NVIDIA markets tokens-per-megawatt as a factory benchmark. An inference-workload-aware IaaS sits exactly between raw GPU hours and token APIs — capacity with workload-class-aware SLOs, energy guarantees, and telemetry — which is differentiation the commodity GPU-hour market does not provide and which complements rather than cannibalizes inference-provider customers, because it makes their factories more efficient at the thing they sell.

Where the evidence disagrees

The sources disagree on three axes, and the disagreements are definitional rather than factual. First, market size: Gartner’s $2.59T (procurement), IDC’s $487B (hardware shipments), Menlo’s $37B (enterprise GenAI purchases, explicitly excluding chips and inference serving), and Bloomberg Intelligence’s $1.8T-by-2032 (revenue potential) all measure different flows; quoting them side by side without the definitions is how “AI is a bubble” and “AI is underfunded” arguments both get made. Second, the training/inference crossover date: Gartner says inference has already passed training in AI-optimized IaaS (2026), Bloomberg Intelligence says 2029 for the wider market, and Epoch’s OpenAI datapoint shows training still dominant at the frontier labs (2024) — the resolution is that the crossover happens first in the commercial cloud, last among the labs that train frontier models. Third, fine-tuning’s importance: the methodology literature treats it as a first-class adaptation technique ¹⁸¹⁹, while the adoption evidence treats it as niche ²⁰ and no market sizing exists — the two literatures are not actually in conflict, they are measuring capability vs deployment.

Gaps and open questions

The gaps that matter: (1) no standalone fine-tuning market size exists anywhere in the fetched evidence — an analyst that publishes one would be filling a real hole; (2) no public split of training compute by modality exists, and it is decision-relevant for power and silicon planning; (3) GPU utilization by workload class is not systematically measured — the numbers that circulate (30–60% training, 10–40% inference, and a 95%-wasted claim from VentureBeat) could not be traced to a verifiable primary source in this review and are deliberately excluded; (4) inference-vs-training revenue splits are reported by vendors with obvious interests (NVIDIA’s ~40%-inference share of datacenter revenue is press-reported, not in the fetched releases, and is flagged accordingly); (5) several widely quoted figures remain unverifiable even after a dedicated search wave: the Gartner “$112B agentic spend by 2028” figure, Deloitte’s agentic share-of-GenAI-spend claims, and coverage of CoreWeave’s reported 2026 price increase (Inc.com) are all behind bot-walls or absent from primary pages. What would settle these: a vendor-neutral training/inference/fine-tuning revenue taxonomy with published methodology; utilization telemetry disclosed by GPU cloud operators (CoreWeave’s 10-K is a start but reports no utilization); and analyst reconciliation of the four market-size definitions.

Confidence and limitations

Confidence is moderate. The mechanism layer is high-confidence — the systems and methods papers are peer-reviewed or widely replicated, and every load-bearing number in the evidence table was grep-verified against fetched full texts this session. The market-data layer is moderate-confidence: it rests on a handful of analyst houses with different definitions, several key figures are press-reported or secondary (Gartner via trade press, NVIDIA’s inference share, the Stanford 2026 investment figure), and the most-cited long-run numbers (BI 2032, Goldman 2030 tokens) are model outputs, not observations. The strategic layer is the most interpretive: the neutrality argument is grounded in primary documents (NVIDIA blogs, CoreWeave 10-K, Sequoia) but the synthesis is this review’s own. Limitations: four remembered landmark titles could not be resolved and were dropped; direct search engines (Google, Bing, DuckDuckGo, Mojeek, Startpage) bot-blocked the research host, so the search wave relied on Google News RSS, publisher WordPress APIs, and direct PDF retrieval — some items that likely exist (VentureBeat’s GPU-utilization analysis, Inc.com’s CoreWeave pricing story, the Gartner and Deloitte agentic-spend figures) could not be read and are listed as gaps rather than cited; SemiAnalysis’s detailed inference-economics models are paywalled and represented only through NVIDIA’s own reporting of the InferenceMAX benchmark and its public article titles; Anthropic’s “State of AI Infrastructure” report could not be retrieved (the URL no longer resolves and searches surface no canonical version); CoreWeave margins are presented as computed from the 10-K income statement, not as reported margins; and the review is a snapshot as of 2026-08-19 — the crossover numbers will keep moving.