Ask a GPU broker what a card costs and you get dollars per hour. Ask an API vendor what a token costs and you get dollars per million. Both numbers are honest and neither one converts into the other without knowing something about the workload — specifically, how many bytes the machine has to move between memory and compute for every token it produces. On transformer inference that number is brutal and undramatic: one token requires reading the entire set of weights. Everything else in this article follows from that sentence.
The one ratio that decides throughput
Inference has two phases with opposite bottlenecks. Prefill ingests the prompt, and because all the prompt tokens are available at once it runs as large matrix multiplications with high arithmetic intensity — hundreds of FLOPs per byte, comfortably compute-bound. Decode produces the answer one token at a time, each token depending on the previous one, so the only parallelism left is across users. Decode has an arithmetic intensity of about 1 FLOP per byte in 16-bit precision: two operations per parameter, two bytes per parameter. Hardware, meanwhile, is built for the opposite regime. An H200 delivers roughly 989 TFLOPS of BF16 dense compute against 4.8 TB/s of memory bandwidth, which puts its balance point — the ridge of the roofline — near 206 FLOP/byte. Decode sits two orders of magnitude below it, so the compute units spend most of their time waiting for data.
The practical consequence is a formula short enough to memorise: tokens per second ≈ memory bandwidth ÷ bytes read per token. For a 70B model stored in FP8 that is 70 GB of weights per token, so a 3.35 TB/s H100 ceilings near 48 tokens/second, a 4.8 TB/s H200 near 69, and an 8 TB/s B200 near 114 — at batch size one, with no KV-cache pressure and no overhead. Those are ceilings, not forecasts; real serving stacks land well below them. But the ordering is what matters, and it is not the ordering the marketing materials suggest: the H200 has identical compute to the H100 and roughly 43% more tokens per second on this workload, purely because it moves bytes faster.
What the spec sheet is actually selling you
Read the current NVIDIA data-centre line as a memory story and it becomes coherent. The H100 SXM carries 80 GB of HBM3 at 3.35 TB/s. The H200 keeps the same compute and moves to 141 GB of HBM3E at 4.8 TB/s. The B200 jumps to 192 GB of HBM3E at 8 TB/s with a 1,000 W envelope, and B300 pushes capacity to 288 GB. The generational gain that actually shows up in serving economics is bandwidth per dollar, and the capacity number is what decides whether a model fits on one card, four, or eight — a distinction that changes your interconnect bill as much as your compute bill.
HBM4, now shipping, is a good illustration of how to read these transitions. JEDEC's JESD270-4, published in April 2025, doubles the interface from 1,024 to 2,048 bits and doubles independent channels per stack from 16 to 32, with a baseline pin rate of 8 Gb/s — which works out to 2.048 TB/s per stack at the standard's floor. Suppliers then overclock well past it: Micron rates its HBM4 above 2.8 TB/s per stack, Samsung quotes 11.7 Gb/s steady with headroom to 13, and NVIDIA's Rubin specifies 288 GB of HBM4 at 22 TB/s per GPU. The detail worth noticing is capacity: the first 12-high HBM4 parts ship at 36 GB per stack, the same as 12-high HBM3E. The generation's first gift is bandwidth, not capacity. Stacks get taller later — 16-high parts reach the standard's 64 GB ceiling — and HBM4E is already being previewed at 4 TB/s per stack.
Why memory is the part that stays expensive
Compute capacity can be added by anyone with a fab slot and eighteen months. HBM is harder for a physical reason that fits in one sentence: a stacked HBM die consumes roughly three times the wafer area of a commodity DDR5 die for the same number of bits, according to disclosures from SK hynix and Micron, with third-party estimates running from three to four times depending on generation. Stacking, through-silicon vias and yield loss all take a bite. So every wafer that goes to HBM removes three wafers' worth of ordinary DRAM from the market — and since AI accelerators are absorbing a growing share of leading-edge DRAM output, the squeeze shows up in server DIMM prices as well as accelerator lead times.
The supply side is also unusually concentrated. Three firms make commercial HBM — SK hynix, Samsung and Micron — and TrendForce's 2026 estimates put the split at roughly 50% / 28% / 22%, with SK hynix's share narrowing as Samsung's HBM4 qualification lands. All three were reported to have sold out their 2026 HBM capacity, and HBM3E allocations were described as effectively sold out into 2027 before HBM4 ramped. That is not a market where a new entrant can relieve pressure next quarter; the fabs that would do it arrive in 2027 and 2028.
This is the asymmetry that matters for anyone buying compute. If GPUs are scarce, you can substitute — older silicon, smaller cards, a different region, a spot market. If memory per accelerator is the constraint, substitution stops working: a model that does not fit needs more cards, and more cards means more interconnect, more failure surface and a worse tokens-per-dollar curve. Memory scarcity does not raise your hourly rate so much as it changes the shape of the deployment you are allowed to build.
What this means if you are the one paying
Stop comparing dollars per hour; compare dollars per million output tokens at your concurrency. Take the hourly rate, divide by the tokens per second the configuration actually sustains, and the ranking often reverses. A card at $2.50/hour producing 48 tokens/second is about $14.50 per million output tokens. The same card under continuous batching, sustaining 2,000 tokens/second across many users, is about $0.35. Nothing about the hardware changed. The difference is that batching reuses each weight fetch across many sequences, which raises arithmetic intensity and drags decode back toward the compute-bound regime it was never designed to be in.
Which is the honest way to read the price gap between a hyperscaler API and a cheap single-tenant GPU rental. The rental is not overcharging for the card; it is selling you batch-size-one throughput, because you are the only tenant. A provider running a busy multi-tenant endpoint can charge less per token and still make more per GPU-hour. Fragmented capacity has a structural penalty here, and it is worth stating plainly rather than pretending the arbitrage is free: a pool of individually rented GPUs pays the memory-bound tax on every token, and that tax is set by HBM bandwidth, not by anyone's business model.
Quantization is a bandwidth optimisation wearing a capacity costume. The usual framing is that FP8 or FP4 lets a bigger model fit. True, but the larger effect on a memory-bound workload is that halving the bytes per weight roughly doubles tokens per second, because throughput is bandwidth divided by bytes moved. That is why every serving stack now ships quantized checkpoints and why pre-quantized weights beat on-the-fly conversion in published benchmarks — the arithmetic is unchanged, the traffic is not. It is also why a 4-bit model can feel faster on a laptop than a 16-bit one on a server with an older memory bus.
Ask what happens at longer context. The KV cache competes with weights for the same HBM. At short sequences the weights dominate and the formula above holds; at long sequences the cache becomes the denominator, and concurrency collapses because there is no room left to hold another user's context. A configuration that serves 200 concurrent short chats may serve a dozen long ones, and the cost per token moves accordingly. Any capacity plan that quotes a single tokens-per-second figure without a sequence length attached is quoting an incomplete number.
The 2026 reality check
Two things are true at once. HBM4 is a real step — double the interface width, meaningfully better power efficiency per bit, and a 22 TB/s memory subsystem on Rubin that will make today's bandwidth ceilings look quaint. And none of it relieves the shortage on the timeline buyers care about: 2026 output is committed, 12-high HBM4 ships at HBM3E capacities, and the wafer trade-off that starves commodity DRAM is a property of the technology, not of this year's allocation. If your plan assumes memory gets cheap in 2027, you are assuming a fab comes online and yields well on schedule.
One deliberate omission. We are not publishing HBM stack prices here. They are negotiated contract prices, they differ by customer and volume, and the public figures we could find disagreed with each other by wide margins. Quoting one would look precise and be wrong. The ratios above — wafer area per bit, bandwidth per stack, capacity per part — are what the purchasing decision actually turns on, and they are stable enough to plan against.
Comments