Every post in this series has circled the same question from a different side: what actually limits an AI cluster? We found the GPU, the HBM stack, the interposer, the rack's heat, the substation, the water at the plant, the battery room behind the transfer switch, and finally the fraction of rented accelerators that sit idle. This one goes after the layer all of those depend on and none of them mentions: the network that turns a pile of servers into one machine. It is the least glamorous line item on an AI cluster's bill and, in a training job, the one that decides the invoice.
The reason is structural. A single GPU generating tokens is a self-contained device — nine tenths of the inference bill happens inside it. A thousand GPUs training a model are not a thousand devices; they are one device that argues. Every gradient all-reduce, every expert routing decision, every pipeline stage handoff is a conversation, and the conversation runs at whatever the slowest link in the room allows. When people quote model FLOPs utilization of 40% on a large training run, the missing 60% is not all silicon: a meaningful share of it is GPUs waiting for bytes.
Two networks, one 9x cliff
Scale-up is the network inside the rack, and it is deliberately absurd. NVIDIA's fifth-generation NVLink gives each Blackwell GPU 18 links of 100 GB/s, for 1.8 TB/s bidirectional — the company's own technical writeup puts that at over 14x the bandwidth of PCIe Gen5. Fifty-two such GPUs in a GB200 NVL72 become a single NVLink domain with 130 TB/s of aggregate GPU-to-GPU bandwidth, which is the hardware behind the "the rack is one GPU" claim; the same documentation describes NVLink domains scaling to 576 GPUs at roughly 1 PB/s of aggregate bandwidth. That is what it costs to make tensor parallelism behave like it lives on one die.
Scale-out is the network between racks, and it is a different product category. Each Blackwell GPU leaves the rack through a ConnectX-8 SuperNIC at 800 Gb/s. In the units that matter for a collective, that is 100 GB/s in each direction — about 9x less than the 900 GB/s per direction NVLink offers. This is not a cost-cutting oversight; it is physics plus economics. Copper reaches a couple of metres at these rates, optics cost money and burn power, and a fat-tree that gives every GPU a full 1.8 TB/s of cross-rack bandwidth would cost more than the GPUs. So the industry picked a number: 400G per GPU in the Hopper generation, 800G per GPU in Blackwell Ultra, and a design pattern — keep as much traffic as possible inside the rack — that every scheduler now assumes.
The practical consequence is the thing worth internalizing: the fabric does not slow your cluster down proportionally, it slows it down where the topology is thin. Tensor-parallel layers that fit inside an NVLink domain run at NVLink speed. Pipeline stages, data-parallel gradient exchanges and expert-parallel routing cross the 9x cliff, and their cost is set by how many of those crossings your parallelism strategy creates. That is why "we have 8,000 GPUs" says almost nothing on its own, and why the same 8,000 GPUs can deliver wildly different throughput per dollar depending on whether the model was sharded to keep hot traffic rack-local.
Radix math: your switch's port count is your cluster's ceiling
Cluster size is not limited by GPUs, racks or power — it is limited by how many ports a switch chip has. A non-blocking two-tier fabric (leaf switches connected to spine switches, half the ports down, half up) supports roughly N²/2 endpoints for an N-port switch; add a third tier and the exponent goes up again. Run the numbers on 2026 shipping silicon and a pattern appears.
NVIDIA's Quantum-X800 InfiniBand switch carries 144 ports at 800 Gb/s — 115 Tb/s of switching capacity in one box, which the company's own material describes alongside hardware SHARPv4, adaptive routing and telemetry-based congestion control. At 144 ports, a two-tier non-blocking fabric tops out near 10,400 GPUs. Broadcom's Tomahawk 6 is the other end of the same curve: 102.4 Tb/s in a single chip, which at 200 Gb/s per link is 512 ports, and 512²/2 ≈ 131,000 endpoints — consistent with Broadcom's own claim of "100,000+ XPUs in a two-tier scale-out network at 200 Gbps/link" and its stated 512-XPU scale-up domain. Higher radix is not a spec-sheet vanity number; it is the difference between needing two tiers and needing three, and every tier adds switches, optics, latency and failure domains.
Now the part that never appears in a GPU cloud's pricing page. In that two-tier fabric, each endpoint consumes one downlink and one uplink — two links per GPU. For 10,400 GPUs that is about 20,800 links, each terminated at both ends, plus the switch count that goes with it (roughly 144 leaves and 72 spines at 144-port radix, by our arithmetic). Inside a rack those links can be copper; across rows they must be optics, and optics are where the network's power and failure budget goes. When a provider quotes you $/GPU-hour, some of that number is transceivers — and when a provider quotes a suspiciously low number, the first thing to check is whether the fabric is oversubscribed, because a 3:1 oversubscribed fabric is a cheaper network that shows up as slower training, not as a line item.
Why Ethernet became credible here — and why InfiniBand did not go away
For most of the last decade the answer to "what fabric for AI training" was InfiniBand, for one reason: it is lossless by design, with credit-based flow control, so a congested link applies backpressure instead of dropping packets. Ethernet's answer was to bolt losslessness on top — priority flow control, ECN marking, RoCEv2 — and tune it, which worked, but made the network an operational project rather than a purchase.
2025 was the year that changed on paper. The Ultra Ethernet Consortium released Specification 1.0 on June 11, 2025, a full stack spanning transport, congestion control, link and physical layers, explicitly aimed at multi-vendor interoperability rather than one vendor's dialect of RDMA. Two weeks earlier, on June 3, 2025, Broadcom announced it was shipping Tomahawk 6 at 102.4 Tb/s — the first single chip past 100 Tb/s — with UEC compliance, support for Clos, rail-only, rail-optimized and torus topologies, and a co-packaged-optics variant. NVIDIA, which sells the incumbent, now sells both: Quantum-X800 for InfiniBand and Spectrum-X at 800 Gb/s per port for Ethernet, the latter with a proprietary claim worth flagging as a vendor claim rather than a measurement — 95% data throughput at scale versus roughly 60% for standard Ethernet, attributed to congestion control and adaptive routing rather than to the wire.
Two projections are frequently quoted as if they were results, and both deserve their labels. Dell'Oro Group projects that Ethernet will surpass InfiniBand in AI back-end networks by 2027 (its January 2025 AI networks report, cited by Nokia among others). 451 Research expects more than 40% of AI-enabled data centers to adopt Ethernet-based AI fabrics by 2027. Both are analyst forecasts about a market that is being decided cluster by cluster, and both are directionally consistent with what vendors are shipping — but a forecast is not a deployment. The honest 2026 statement is narrower and more useful: the protocol argument has stopped being a capability argument and become an operations argument. If your team already runs Ethernet with a mature automation stack, a UEC-shaped fabric is now a defensible choice; if your job is a single very large training run and you can afford the specialty, InfiniBand's maturity still buys something real.
The optics bill, and why the topology moved it
The quiet architectural change in AI data centers is that the first tier of switching left the rack. In a CPU-era hall, top-of-rack switches sat beside the servers and copper did the work. In an AI hall the leaf switch moved to the end of the row, because the compute trays are too dense and too liquid-cooled to host it — and that move converts short copper runs into long optical ones, at both the NIC end and the switch end. NVIDIA's own engineering writeup makes exactly this argument: the topology shift is what makes optics mandatory rather than optional, and therefore what makes optical power a first-order cluster cost.
Which is why co-packaged optics became the 2026 battleground. Putting the optical engine in the same package as the switch removes the pluggable module, the electrical path to it, and a chunk of the signal conditioning. NVIDIA claims its photonics switches deliver 3.5x better power efficiency and 10x better resiliency than traditional pluggable designs, and quantifies the mechanism as reducing electrical signal loss from about 22 dB to roughly 4 dB. Broadcom's third-generation CPO part, Tomahawk 6 – Davisson, claims a 70% reduction in optical interconnect power — "more than 3.5x lower than traditional pluggable solutions" — at 102.4 Tb/s of optically enabled switching. Treat all three as vendor claims from parties selling the alternative, and treat the direction as settled anyway: at 800G per port and two links per GPU, optics are no longer a rounding error on a cluster's power budget, and the companies that make both the switch and the laser are competing on exactly that line.
What to ask before you rent or build
If you rent GPU capacity, the fabric is the part of the product you cannot see and the part most likely to be quietly degraded. Five questions cover it. One: what is the per-GPU scale-out bandwidth — 400G, 800G, or PCIe-attached afterthought? Two: what is the oversubscription ratio between tiers, and is the advertised number the bisection bandwidth or the marketing one? Three: is the fabric InfiniBand or Ethernet, and if Ethernet, is it UEC-shaped or hand-tuned RoCE with a single engineer who understands the ECN config? Four: is the fabric isolated per tenant, or shared with neighbours whose incast patterns will show up as your tail latency? Five: what topology — rail-optimized, Clos, torus — and does the scheduler know it?
If you are building, the arithmetic from earlier is the whole design review: pick your switch radix, compute N²/2, decide whether that number is above or below your target cluster, and count two links per GPU before you approve the optics BOM. A cluster that is one tier short is not a cluster that runs 10% slower; it is a cluster whose largest jobs do not fit at all. And if you are running a single GPU at home — the home-lab TCO case — your east-west traffic bill is exactly zero, which is the structural reason one machine is superb at inference and hopeless at training. The fabric is what you buy when you stop being one machine.
Worth saying plainly for anyone sizing a distributed network of smaller nodes rather than a single monolith: the 9x cliff is the argument for both architectures at once. Rack-scale domains win on training because they keep the hot traffic on NVLink; geographically spread capacity wins on inference because inference has almost no east-west traffic to lose. Distributed compute networks inherit the second case and inherit its weakness — a node without adequate interconnect is fine for a request and useless for a gradient (omc.network). Neither architecture escapes the arithmetic; they just sit on different sides of the cliff.
The wider point is the one this series keeps arriving at from new directions. In 2026 the constraint on AI compute is a stack of coupled ceilings — power, package, memory, water, utilization — and the network is the ceiling that is easiest to hide, because it has no spec sheet on a purchase order. GPUs are advertised in units of GPUs. Clusters should be advertised in units of bisection bandwidth per GPU, and almost none of them are.
Comments