There is a category of cost in AI infrastructure that survives every architecture argument: work you did once, lost because a component failed, and work you did again to get back to where you were. It is not captured by FLOPS, by utilization percentages, or by the $/GPU-hour on a rental page. It is captured by two numbers that most operators never publish — how often their cluster drops a job, and how long it takes to write and reload the model's entire state.

Meta published both, indirectly, in the Llama 3 paper. The 405B pretraining run used 16,384 H100 80GB GPUs and, over a 54-day snapshot, logged 466 job interruptions: 47 planned (firmware upgrades, operator-initiated config changes) and 419 unexpected. That is 7.8 unplanned interruptions per day, or one every 3.1 hours cluster-wide by our division — the number Data Center Dynamics headlined as "one failure every three hours". The paper's own framing is worth keeping: the synchronous nature of training means a single GPU failure may require restarting the entire job.

What breaks, measured

The root-cause table is the most useful artifact in the paper for anyone buying compute, because it is a public fleet-scale failure census for H100-class hardware. Of the 419 unexpected interruptions, faulty GPUs accounted for 148 (30.1%) and GPU HBM3 memory for 72 (17.2%) — together more than half. Software bugs were 54 (12.9%), network switches and cables 35 (8.4%), unplanned host maintenance 32 (7.6%), GPU SRAM 19 (4.5%) and the GPU system processor 17 (4.1%). The rest is a long tail in single digits: NICs, NCCL watchdog timeouts, silent data corruption, thermal-interface and sensor faults, SSDs, power supplies. Meta attributes about 78% of all unexpected interruptions to confirmed or suspected hardware.

Two details in that table should change how you think about redundancy. First, HBM is the second-largest single cause and it is not a "GPU died" event — memory faults are often silent until they are not, which is why the same table lists six cases of silent data corruption. A cluster that only monitors "is the GPU present" misses a fifth of its interruptions. Second, the whole category of switches and cables (8.4%) is larger than every host component combined: CPU, system memory, SSD and power supply together account for roughly ten events in 54 days. The failures cluster where the bandwidth is, which is consistent with the fabric post in this series — the network is not a passive cable, it is a component with its own failure rate.

The historical comparison is the one everyone quotes for a reason. OPT-175B, trained on 992 A100s in 2022, recorded "at least 35 manual restarts and the cycling of over 100 hosts over the course of 2 months", with the authors estimating 70+ automatic restarts on top. Two years and one hardware generation later, Llama 3 had 16.5x the GPUs and lost three interruptions to manual intervention, with the rest handled by automation. The failure count went up with scale; the human cost went down. That is the actual progress in the last three years of AI infrastructure, and it is invisible in any GPU spec sheet.

A checkpoint is 16 bytes per parameter

Reloading from "the last saved checkpoint" only works if saving is cheap enough to do often, so the first number to compute is how big the thing is. Mixed-precision Adam is the standard recipe and it multiplies the parameter count by a constant that surprises people: 2 bytes for the bf16 weight, 2 for the bf16 gradient, 4 for the fp32 master copy the optimizer actually updates, and 4 + 4 for Adam's two moments — 16 bytes per parameter, or 112 GB for a 7B model. For 405B parameters that is 6.48 TB per full checkpoint, before activations, before any sharded replication, and before the two or three previous checkpoints you keep so a corrupted save does not end the run.

Now convert that into the only unit that matters operationally: bandwidth. Writing 6.48 TB in 30 seconds requires 216 GB/s of aggregate write throughput. Spread across 16,384 GPUs that is a modest 13.2 MB/s per GPU — the per-GPU number is trivially small, which is exactly why people assume checkpointing is free. It is not free, because 216 GB/s is a storage cluster problem, not a node problem: it is roughly the aggregate bandwidth of several thousand NVMe drives behind a parallel filesystem, and it has to be provisioned for a burst that happens every fifteen minutes and lasts half a minute. Storage sized for the average is storage that stalls the most expensive computers on earth.

Which is why every serious stack does the same two things. It writes asynchronously: the GPUs dump state to host memory in a fraction of the time, training resumes, and the write to the parallel filesystem completes in the background — the technique every current framework advertises, and the reason published save costs are measured in tens of seconds rather than minutes. And it reloads fast: an operational study of a 504-GPU pretraining cluster published this year measures checkpoint save durations of 18.0 s, 31.7 s and 30.0 s across its three training phases by counting consecutive NFS write-spike samples — independent confirmation that a small cluster's save cost lands in the same tens-of-seconds range that the arithmetic above assumes for a much larger one.

The interval is an equation, and it has an answer

Given a save cost and a failure rate, how often should you save? This was solved for supercomputers decades ago and the answer — the Young/Daly result — is now standard in LLM infrastructure papers: the optimal interval is √(2 × δ × M), where δ is the cost of one checkpoint and M is the mean time between failures. At that interval the time spent saving equals the time spent re-doing work, and the minimum fraction of wall clock lost is √(2δ/M). Everything about checkpoint strategy follows from that square root, including its mercy: because the loss scales as the root, a failure rate a hundred times worse only costs ten times the overhead.

Plug in Meta's fleet. M = 3.09 hours = 11,124 seconds; a well-engineered 405B save costs δ = 30 seconds. The optimal interval is √(2 × 30 × 11,124) = √667,440 ≈ 817 seconds, or about 13.6 minutes. At that cadence the run spends 30/817 ≈ 3.7% of wall clock writing checkpoints and 817/(2 × 11,124) ≈ 3.7% re-doing work after a failure — a 7.3% floor that no amount of engineering removes, only trades between the two terms. Snapshot 405B pretraining at 21.2 million GPU-hours (16,384 GPUs × 1,296 hours, our multiplication) and 7.3% is 1.55 million GPU-hours spent on saving and re-doing rather than learning. Price that at whatever $/GPU-hour you rent at; the point is that it is not a rounding error.

The reason this is worth stating as arithmetic rather than anecdote is that the same equation explains why most real clusters are misconfigured in the same direction. In that 504-GPU study, the operator's actual intervals were 133.5, 199.0 and 81.5 minutes against theoretical optima of 44.9, 59.7 and 58.1 minutes — roughly 3x too sparse in two phases and 1.4x in the third. The measured cost of that conservatism: total overhead of 2.20%, 3.22% and 1.82% of wall clock, with save overhead itself never exceeding 0.61%. Their conclusion is the counterintuitive one worth stealing: because a save is cheap, saving more often is nearly free, and the expensive mistake is saving too rarely. Their mean lost time per abnormal termination was 0.98 hours across 23 events — about an hour of 504 GPUs, thrown away and recomputed, per crash.

The checkpoint is also a power event

There is a second bill, and it lands on the utility rather than the storage vendor. Meta's paper describes what happens when tens of thousands of GPUs stop together: "when tens of thousands of GPUs may increase or decrease power consumption at the same time, for example, due to all GPUs waiting for checkpointing or collective communications to finish, or the startup or shutdown of the entire training job... it can result in instant fluctuations of power consumption across the datacenter on the order of tens of megawatts, stretching the limits of the power grid." Checkpointing appears in that sentence as a cause of a grid event, on equal footing with starting a job.

The workaround the industry converged on is telling: SemiAnalysis, reading the same disclosure alongside a 2025 Microsoft/OpenAI/NVIDIA paper on power stabilization, reports that engineers built a switch literally named pytorch_no_powerplant_blowup=1 that injects dummy work to smooth the draw — and that at gigawatt scale the wasted energy from that smoothing runs into tens of millions of dollars a year. The same source puts Meta's 24,000-H100 cluster at roughly 30 MW of IT capacity, which is the scale at which "save the model" becomes a substation problem. This is the same coupling the backup-power post in this series described from the other end: the building is sized for a load that changes in under a second, and the training loop is what changes it.

Five questions for anyone selling you time on a cluster

If you rent long-running training capacity — reserved, spot, or "we'll schedule it for you" — the checkpoint regime is part of the product, and it is usually undisclosed. One: what is the cluster's job-stopping MTBF, measured rather than marketed, and over what fleet size? A number without a denominator is a slogan. Two: what is the checkpoint save cost for a model of my size, and is it synchronous or host-memory-buffered? Three: what aggregate write bandwidth does the storage tier actually deliver during a burst, and is it shared with other tenants who checkpoint on the same schedule? Four: what is the preemption notice on spot capacity, and does my restart pay a full re-load or a warm one? This is where the reserved-versus-spot spread is really decided — a cheap preemptible GPU with a 20-minute reload is not cheap. Five: how is "effective training time" defined, and does it include the re-done work or only the queued-and-running time?

The last question matters because the headline number flatters everyone. Meta's "over 90% effective training time" is the good outcome of this arithmetic — it sits just above the 7.3% floor that the equation says is unavoidable at a 3-hour MTBF, with the remainder absorbed by fast diagnosis and automation. If a provider quotes 99% effective training time on a 16k-GPU cluster, either their MTBF is far better than Meta's published fleet, or the definition excludes something. Ask which.

For anyone building on rented or distributed capacity rather than a single owned cluster, the arithmetic gets harsher before it gets better: smaller gangs have shorter MTBFs for the same per-node failure rate, and a restart that loses an hour is an hour you paid for twice — which is why the honest pitch for geographically spread compute is inference, where a request either completes or retries in milliseconds, not training (omc.network). Neither architecture escapes √(2δM); they just sit on different sides of the equality. The utilization tax in this series is the same phenomenon measured from the other direction: most rented accelerators run at 30-50% of nameplate, and a meaningful share of the missing half is not idle GPUs but GPUs waiting for a failed job to be rebuilt from disk.

The wider point is the one this series keeps arriving at from new directions. In 2026 the constraint on AI compute is a stack of coupled ceilings — power, package, memory, network, water, and now the reliability of the fleet itself — and the checkpoint is where three of them meet: it consumes storage bandwidth, it costs GPU-hours, and when it fires it moves tens of megawatts. Nobody puts it on a purchase order. Everybody pays for it.