This forum has a habit of finding the AI bottleneck one layer at a time. First GPUs, then HBM, then substations, then the building, then the package, then the water at the power plant. Every one of those is real. But they share a quiet common denominator that none of them names: a chip that exists, is powered, and is paid for is still not the same thing as a chip that is doing work. The gap between the two is the cheapest, largest, most ignored supply of AI compute there is.
This piece does the same thing the others did: separate two numbers that get quoted as one, find where the difference actually comes from, and work out what it does to the price you pay.
What "utilization" actually measures — and why two numbers get confused
The word gets used for at least three different things, and mixing them is how a 2× error enters a capacity plan. GPU utilization (often the SM-occupancy or DCGM metric) is the share of the chip's execution units that are busy at a sampled instant — commonly reported in the 50–70% range for actively running jobs. Model FLOPs utilization (MFU) is the harder number: actual floating-point ops delivered divided by the chip's theoretical peak, over a whole training run. Because of memory-bound phases and overhead, MFU is always far below the SM-occupancy number, and it is the one that decides your token throughput. Then there is slot occupancy — what fraction of reserved capacity is even running a job at all, before you ask how hard the chip inside is working.
A cluster can show 65% SM occupancy, 40% MFU, and 55% slot occupancy at the same time, and all three are true. Quoting the 65% as "how utilized we are" is the single most common way utilization gets oversold. The number that matters for supply is the product of all three, and it lands well under half.
Three places the tax is collected
The first is burstiness. Training and inference are not smooth rivers; they are waves. A reserved cluster sits idle between experiments, between checkpoints, between a batch finishing and the next one being staged. Reservation-based billing means you pay for the wave even during the trough. This is the largest, most structural leak, and it is pure scheduling — not physics.
The second is memory-bound math. A modern accelerator does on the order of 1014 FLOP/s but moves bytes at a fraction of that. Decode — generating one token at a time — touches memory roughly once per FLOP, so it runs near ~1 FLOP per byte against a hardware balance point near ~200. The arithmetic units are literally waiting on HBM for most of every millisecond. This is the same wall the HBM deep-dive described, now expressed as idle ALUs rather than as a line item.
The third is scale-up overhead. The moment you train across hundreds or thousands of chips, you pay a coordination tax: all-reduce synchronizations, pipeline bubbles from imperfect micro-batching, and checkpoint stalls. Multiple published scaling studies show per-device efficiency falling as cluster size grows, because the communication and bubble cost grows faster than the extra compute helps. Bigger is not proportionally better, and the gap is paid in utilization.
Why the headline "shortage" is mislabeled
When an executive says "we cannot get enough GPUs," translate it carefully. A fleet billed at 100% but delivering 40% MFU is, for the purpose of tokens produced, a fleet that is 60% absent. Saying "we need more GPUs" when 60% of the existing ones are not producing is like saying "we need a bigger warehouse" when half the floor is empty. The constraint is used GPU-time, and used GPU-time is a function of software and scheduling, not of silicon.
This is why the same model that "needs" 25,000 H100s on one accounting can often be served with fewer, better-utilized devices on another. The difference is not the model. It is the tax.
What this means for the price you pay
Rent is quoted in $/GPU-hour, but value is delivered in $/token or $/finished run. The two diverge exactly by the utilization tax. A 40% MFU cluster charges you for 100% of the silicon and gives you 40% of the math; your effective $/token is 2.5× the theoretical floor. Two providers quoting the same $/hour can be 2× apart in $/token depending on how they schedule, and almost none of them print the MFU on the invoice.
For buyers, the practical move is unglamorous: ask for the utilization, not the rate. A lower $/hour on a 30%-MFU reservation can be a worse deal than a higher $/hour on a 50%-MFU one. The discount is in the utilization, and the utilization is the number nobody volunteers.
The distributed inventory nobody has booked
Here is where the editorial position of this site becomes relevant, and we state it plainly as a disclosure rather than sneaking it in. A decentralized GPU network's entire premise is that the shortage is one of coordination, not of hardware: there are millions of capable GPUs sitting in workstations, offices, gaming rigs and small mining rigs, used a few hours a day and idle the rest. Booking that idle time — already powered, already amortized, already cooled — is supply that requires no new fab, no new substation, no new packaging line. It does not replace frontier training clusters; it is the obvious place for the bursty, latency-tolerant, throughput-hungry middle of the market that is currently paying the full utilization tax on reserved hyperscale capacity.
The catch is real and worth naming: distributed supply is heterogeneous, harder to schedule, and higher-latency than a co-resident cluster. Pretending otherwise would be the kind of claim this forum exists to puncture. The point is narrower and stronger — the utilization tax proves there is far more usable compute than the "GPU shortage" headline implies, and the cheapest megawatts of AI capacity to unlock are the ones already humming in someone's office.
The 2026 reality check
State the soft spots. The single most important one: MFU and "GPU utilization" are different metrics and are routinely quoted as if they were the same, so any utilization claim needs the exact definition attached. Fleet-telemetry samples vary by who is measuring, what workloads are in the mix, and whether idle-but-reserved time counts as "used"; the 30–40% datacenter-wide figures trace to a small number of telemetry vendors and should be read as estimates, not measurements. The PaLM ~46–47% MFU is a real disclosure but it is TPUs and one model family, so it anchors the top of the band rather than defining it. What survives all the caveating is the structure, and it is robust: utilization is structurally far below 100%, the gap is large (often more than half), it is collected in predictable places, and it is the cheapest supply to unlock because it needs scheduling and software, not concrete. Re-check the current figures before you price anything, and when a provider quotes utilization, ask which of the three definitions they mean.
Comments