What are the margins on inference?

By Ganeshkumar Ashokavardhanan · data as of Oct 25, 2026

Source

One GB200 NVL72 rack serving GLM-5.3 for a day at 95 tokens/s per user.

$20,676revenue1.1B out · 67.2B in
$10,109rent$421/hr 3-yr contract
$414overhead2% of revenue
$10,153profit49% margin
Profit per day as the rack gets busier
−$20K−$10K$0$10K$20K$30K0%25%50%75%100%GPU utilization58%32%16%own3-yr contracton demand
At 65% utilization: $10,153 a day renting on a 3-yr contract. Break-even at 32%.

Each GPU sells 171 output tokens/s at $19.39 per million, input included: $11.97 an hour against $5.85 rent. Each user sees 95 tok/s, what OpenRouter calls throughput.

break-even $844/hr · last public 3-yr contract: $261/hr (HIVE, Jun 2026); newer deals price higher · how rent is set

break-even 32% · share of the rack's measured throughput you sell · why 65% by default

Token price
$1.12 in · $0.14 cached · $4.40 out, per million

profitable across the range · OpenRouter's GLM-5.3 page, 30 days: 63 : 1 (35 to 85 by day) · how it is worked out

profitable across the range · OpenRouter, all providers: 90% (86% to 94% by day) · each point is about 6% of capacity · why it matters so much

break-even 51% of revenue · on-call engineering, support, billing and the API front end · what is included

break-even 137 tok/s · p90 tokens/s per user; faster per user means fewer users per GPU · why it lowers profit

How the numbers add up

Throughput per GPUoutput tok/s
Benchmark1InferenceX, 95 tok/s per user311
On real trafficOpenRouter: 6.3 fresh input tokens per output vs 5 in the benchmark269
Sold15,1698% uptime × 65% utilization171
Tokens per day
Output1.1B
Fresh input10% of 63 per output6.7B
Cached input290%60.4B
12% of all GLM-5.3 traffic on OpenRouter
Revenueper day
Output31.0661B × $4.40/M$4,691
Fresh input6.7164B × $1.12/M$7,522
Cached input60.447B × $0.14/M$8,463
Total$20,676
Costper day
Rent872 GPUs × $5.85 × 24 h$10,109
Overhead142% of revenue$414
Total$10,522
Profit49% of revenue$10,153

Not included: taxes, and sales and marketing beyond the overhead slider. Totals are computed before rounding.

FAQ

Where does the throughput number come from?
InferenceX's agentic benchmark, which replays 393 Claude Code sessions on GLM-5.2 (the same architecture as GLM-5.3) in NVFP4 on Dynamo SGLang and reports output tokens/s per GPU at each per-user speed. Those sessions carry longer conversations than OpenRouter's average, and longer conversations are slower to serve, so real traffic may run somewhat faster than measured. The receipt then adjusts for OpenRouter's traffic mix, uptime and utilization.
How optimized is the serving setup?

The GB200 figure comes from the 24-prefill, 16-decode run in InferenceX's public recipe (run logs):

  • NVIDIA Dynamo over SGLang, October 2026 development builds
  • prefill on 24 GPUs (three workers of 8), decode on 16 (one worker); throughput is per GPU
  • NVFP4 weights, FP8 KV cache, EAGLE speculative decoding with up to 6 draft tokens
  • experts spread across the 16 decode GPUs, data-parallel attention, FlashInfer MoE kernels
  • sessions pinned to one prefill worker; prefill caches overflow to CPU memory

The page applies that per-GPU rate to all 72 GPUs, as if the rack were split ideally for its traffic (about 47 prefill and 25 decode). With this recipe's 8- and 16-GPU blocks the nearest split, 40 and 32, serves about 15% less. Providers with their own stacks may do better: GLM's GB200 throughput at 100 tok/s per user went from 69 to 287 per GPU between August and October 2026.

Other estimates show 50–90% margins; why is this lower?

Rent explains most of it.

  • They charge about $5 per GB300-hour, roughly what an owner or a large multi-year deal pays; on demand, one rack costs $12.42. At $5 rent and list prices this page gives a GB300 73% at 60% utilization, and its own contract and owning cases are 49% and 73%.
  • This page uses OpenRouter's real traffic, whose 63 input tokens per output load prefill harder than the benchmark sessions, and median provider prices rather than list.
  • DeepSeek's $38 of revenue per GPU-hour comes from a lab serving its own model on its own stack, with capacity that grows with demand.
  • Serving software is improving fast. 25% more tokens per GPU takes the margin from 49% to 59% on the contract and from 10% to 28% on demand.
Where do kernels, speculative decoding and routing show up?

Only in the throughput. The page has no control for serving software; its throughput is one measured run, and a better stack shows up as a higher number. On OpenRouter's traffic a GB200 spends about 65% of its time on prefill (fresh input) and 35% on decode (output), so prefill speed matters more than decode speed here.

  • Prefill gets faster with attention and FP4 GEMM kernels, and with chunk sizes tuned to the context lengths.
  • Decode gets faster with speculative decoding, MoE kernels and wider expert parallelism. The run already drafts up to 6 tokens a step with EAGLE.
  • The page assumes the ideal split between prefill and decode GPUs. A real deployment loses some capacity to worker block sizes unless its split is tuned to its traffic.

The two biggest gains are on the sliders. Cache hits rise when a conversation is routed back to the rack that holds its cache and when caches stay longer in GPU and CPU memory. Utilization rises with batch or off-peak pricing, or a second model on the rack to fill the quiet hours.

Why GLM-5.3?
It is MIT-licensed, among the most used models on OpenRouter, mostly by coding agents, and InferenceX publishes agentic benchmarks for it on GB200 and GB300. The repository also models DeepSeek V4 Pro.
How is "63 input tokens per output" worked out?
OpenRouter's GLM-5.3 page publishes daily prompt, completion and cached token totals across all providers. From Sep 25 to Oct 24, 2026: 16,992B prompt tokens and 271B completion tokens (reasoning included), 62.7 to 1, with 90.3% of prompt tokens read from cache. The calculation and raw data are in the repository.
Why does the cache rate matter so much?
Cached input bills at $0.14 per million instead of $1.12 and skips prefill, which is where most of the GPU time goes on this traffic. A miss means 63 more fresh tokens to prefill per 100 output tokens, so each point of cache hit is about 6% of the rack's capacity. The 90% comes from OpenRouter's daily cached-token counts across all GLM-5.3 providers (86% to 94% by day). It is what providers report in each response, which is also what OpenRouter bills from. Agents resend their context every turn, so hits depend on a conversation's later turns coming back to the rack that holds its cache.
Which prices does it use?
The Median preset is the median of its 16 FP4 endpoints, taken separately for output, fresh input and cached input. Low is DeepInfra's discounted FP4 price; High is Z.ai's list price, which 13 of its 37 providers also charge. Cache writes are free on all but one endpoint, so the page does not charge for them.
Are reasoning tokens counted?
Yes. A reasoning token takes the same GPU work as any other output token, and OpenRouter bills it at the output price. 71% of GLM-5.3 output tokens on OpenRouter are reasoning (63% to 76% day to day): of about 1,067 output tokens per request, about 306 are the visible answer.
What does GPU utilization mean, and why 65%?

The share of the rack's measured capacity that is sold, averaged over the day. A rack has to be ready for its busiest times, so it sells the average share of them:

  • across days, OpenRouter's open-model traffic averages 81–84% of its week's busiest day (calculation)
  • within a day, Azure chat traffic averages 77% of its busiest hour (Azure LLM Inference Dataset 2024; DynamoLLM puts peaks at 1.7× the average for chat, 2.8× for coding); worldwide agent traffic is flatter (FineServe, 2026)
  • a high-traffic model gets the upper end, with headroom for bursts: 0.84 × 0.82 × 0.95 ≈ 65%
  • first-party services run higher (DeepSeek's fleet averaged 82% of its peak); multi-model marketplaces lower (Alibaba's GPUs were 13–34% busy, Aegaeon)
  • a new provider also has to win OpenRouter's routing on price, speed and reliability to get traffic at all

Renting and owning get the same 65%: a single rack cannot hand back part of itself when demand drops.

Why multiply the benchmark by utilization?
InferenceX measures the most the rack can produce while every user still gets 95 tok/s. Utilization is the share of that with paying traffic behind it; lighter demand gives users faster responses at the same price per token. The benchmark's cache accounting can be read two ways, which puts real traffic between 231 and 308 tokens/s per GPU rather than at 269.
Why does faster output per user lower profit?
The slider is p90: the speed the slowest tenth of requests get. A GPU serves many users at once. Faster output for each means fewer users per GPU, so the rack makes fewer tokens in total. Prices stay fixed here; fast tiers on OpenRouter charge about 1.5×.
How is the rent set?
The 3-year contract is the 75th percentile of what 2026 deals are reported at, because new-contract prices rose through the year: $5.85 per GPU-hour for GB200 and $6.60 for GB300. The last contracts with public terms are HIVE's: $261/hr for a GB200 rack over 3 years (Jun 2026) and $285/hr for GB300 over 5 (Aug 2026). GB200 and GB300 racks are hard to buy on demand at all, so the on-demand option uses the cheapest list price as an assumed equivalent: CoreWeave's $756/hr ($42/hr per 4-GPU node, sold as full racks), with AWS at $762 and Oracle at $1,152; for GB300, Verda's $894 and Oracle's $1,296.
Why is owning so much cheaper than renting on demand?
The on-demand price pays for the cloud's margin, its spare capacity and the right to hand the rack back; owning or a contract trades that flexibility for price. An owned rack's price, less 10% resale, is spread over 4 years as equal yearly payments that include 9% a year on the money tied up in it, the same as paying off a 9% loan; support, operations, space and power come on top. Pick Own above to see each line.
What about NVIDIA's "$5M in, $75M of tokens out"?
NVIDIA's InferenceMAX post says a $5M GB200 NVL72 produces $75M of tokens over three years. That needs 5,505 output tokens/s per GPU, sold every second; on agentic traffic InferenceX measures 312 on GB200 at 95 tok/s per user.
What overhead is included, and why 2%?
Billing, customer support, on-call engineering and the API front end, at 2% of revenue by default. About 1% is billing: Stripe charges 2.9% + 30¢ on card payments and 0.4% on invoices, while traffic through OpenRouter costs the provider nothing, because the buyer pays the fee and OpenRouter passes provider prices through without markup. About 1% is people: a 24/7 on-call, support and API team of 20 to 30 spread across a fleet of 100 racks. A smaller fleet spends more, and public companies spend far more: CoreWeave's general and administrative expense was 12.7% of revenue in 2025, with growth hiring and stock compensation the page leaves out. The rack price already covers CPUs, networking and storage. Owning adds $400 a day of hardware operations and 4% of the rack price a year for support and spares. Taxes, marketing and sales are left out.
How current is this?
Prices, traffic and throughput are as of 2026-10-25. They move fast: GLM throughput on GB200 roughly quadrupled between August and October 2026, and new-contract rents rose through the year. Each data snapshot is tagged in the repository.
Can I change the assumptions or contribute?
Every input above is a slider or a toggle, and the link records your settings. Corrections and newer data are welcome as pull requests on GitHub.

Sources (17)

  1. InferenceX AgentX: GLM-5.2/5.3 on GB200/GB300, 393 Claude Code traces, p90 interactivity 2026-10-23
  2. OpenRouter GLM-5.3 page: daily prompt, completion and cached token counts, Sep 25 to Oct 24, 2026 (ratios computed in research/openrouter_ratios.py) 2026-10-25
  3. OpenRouter GLM-5.3 endpoints and prices (48 endpoints, 16 FP4) 2026-10-25
  4. Z.ai API pricing 2026-10-16
  5. CoreWeave pricing: GB200 NVL72 $42.00/hr per 4-GPU node ($10.50/GPU-hr) 2026-10-25
  6. CoreWeave docs: NVL72-powered instances must be deployed as full racks of 18 nodes 2026-10-27
  7. Verda GB300 NVL72 pay-as-you-go $12.42/GPU-hr ($894.20 for 72 GPUs) 2026-10-25
  8. Disclosed NVL72 contracts (HIVE/BUZZ 8-Ks) and 2026 price rise (CoreWeave Q2 call, Nebius Q2 letter, IREN FY26 call) 2026-10-25
  9. SemiAnalysis, H100 vs GB200 NVL72 training benchmarks (capex section) 2024, re-cited 2026
  10. Data Gravity, How much does an NVIDIA NVL72 cost? 2026-08-31
  11. EIA Electric Power Monthly, Table 5.6.A 2026-07 (preliminary)
  12. CBRE North America colocation pricing via Encor Advisors 2025-H2
  13. Owning opex: NVIDIA DGX support terms; CoreWeave 2025 10-K technology & infrastructure expense less D&A (~$1.04M per MW-year, ~$342 per 120 kW rack-day); remote hands $125/hr 2026-10-25
  14. Stripe pricing: 2.9% + 30¢ per domestic card payment, 0.4% per paid invoice, 0.8% for ACH debit (capped at $5) 2026-10-28
  15. InferenceX TCO calculator: mean time between interrupts 24 days, 12-hour recovery, availability 0.98; Meta Prometheus GB200/GB300 MTBI ~47-54K GPU-hours 2026-10-25
  16. Load shape of LLM serving: OpenRouter open-model traffic, average day 81-84% of the busiest day of its week, Sep 25 to Oct 24, 2026 (research/openrouter_load_shape.py); Azure LLM Inference Dataset 2024, chat averages 77% of its busiest hour (research/azure_trace_utilization.py) 2026-10-26
  17. zai-org/GLM-5.3 model card 2026-08-28