How Meta Maintains AI Training Capacity at Scale
How Meta patches thousands of GPU training hosts without pausing time-critical jobs — maintenance trains, gradual rollouts, and the OpsPlanner orchestrator.
Patching a cluster sounds like a solved problem: schedule a maintenance window, take the machines down, bring them back up. At Meta’s scale the window itself is the problem — time-critical training jobs can’t survive one — so maintenance became capacity you reserve on a schedule, not an event you schedule when jobs allow.
Source
Maintaining large-scale AI capacity at Meta
By: Saranyan A. Vigraham, Benjamin Leonhardi (Meta)
Published: June 12, 2024
Type: Engineering article
Build posts are common; keep-it-running posts are rare. This one documents the unglamorous machinery — maintenance trains, sized domains, reserved buffers, an orchestrator — that lets Meta update firmware and drivers across live GPU clusters without breaking the synchronized jobs running on them.
Patching a fleet that can’t pause
The setting: Meta operates many data centers with GPU training clusters worldwide, and a year before this article the AI boom forced a fleet transformation, since generative models now carry trillions of training parameters. At publication, the training infrastructure spanned dozens of clusters running thousands of jobs a day for hundreds of teams, with a plan to reach 600,000 GPUs within the year. Those jobs range from a single GPU running two minutes to workloads spanning thousands of hosts in lockstep — and the latter are extremely sensitive to interruption. The reconfiguration happened without pausing hypergrowth, a task the authors call “akin to rebuilding an airplane mid-flight,” with vendors and utility companies pulled into the effort.
Here “maintaining” means any operation that updates or verifies software and firmware in the clusters — firmware being the low-level code baked into hardware — including the networking path. That definition matters because GPU training is hostile to exactly this kind of work, for five reasons the article names. Many jobs are time-critical, recurring, or online, so capacity can’t be taken down by default. Bad hosts are very bad: a host that’s merely slow, or has a non-fatal hardware quirk, damages a synchronized job disproportionately. Interruptions hurt because thousands of hosts cooperate on one shared problem. The AI software stack is deep, so problems are hard to pinpoint and rollouts must be careful. And cross-host jobs want consistency — outside of CUDA versions there are few hard incompatibilities, but consistent hosts make debugging and avoiding SEVs (Meta’s term for major incidents) far easier.
What makes Meta’s version harder is its own appetite: bespoke training hardware, the newest chips, speed-optimized backend networks, and a deliberately current software stack. The resulting load is more than 30 maintenance operations, more than 50 components being updated, three host-verification tasks, and thousands of disruptive AI host tasks every day — all while guaranteeing capacity. A smaller environment can take a whole cluster into one maintenance window and upgrade everything at once. At Meta’s size and diversity, the authors say, that would be operationally infeasible and risky; instead components are validated for compatibility and rolled out in a sliding fashion, with many overlapping changes in flight at any moment.
The core mechanism is the maintenance train. Picture a train that departs on a fixed schedule and stops at every station: the fleet is divided into maintenance domains — slices of capacity — and a small slice is taken down, fully upgraded, and returned, cyclically. The guarantee is the point: all capacity minus one domain is up 24/7, so capacity is predictable. That guarantee is mandatory for online and recurring training workloads. Every new upgrade gets picked up by a train and completes a full visit of the fleet within a guaranteed timeframe; longer-running upgrades can span multiple cycles, and AI capacity gets specially optimized domains, strict SLOs (service-level objectives), and a contract that lets a service dodge a train stop when possible.
Gradual rollouts handle a subtler problem: during an upgrade, different servers in the same cluster briefly run different host stacks. The design rule that makes this safe is a split — the AI job’s own stack, including the CUDA library it runs with, stays consistent everywhere, while lower-level components like drivers and firmware drift gradually underneath. The reason is timing: low-level pieces can take hours to install, configure, or reboot around, while the job’s container restarts in moments. Getting here took careful low-level testing, special monitoring, tight vendor collaboration, and three years of the stack maturing, plus extra tooling for the rare compatibility-breaking upgrade.
Sizing the domains is treated as an optimization problem, not a guess. A maintenance domain is the percentage of capacity taken down in one go, and its optimal size is a function of two costs: the interruptions you cause training jobs, and the capacity lost during the maintenance window. Meta worked with AI teams on this tradeoff and significantly reduced the maintenance overhead for AI capacity. My example to make it concrete: if a fleet were split into 20 equal domains, the guarantee means at most one is down at any moment — 95 percent always serving — and each domain pays the interruption cost only when the train reaches it.
Holding all of this together is OpsPlanner, Meta’s disruptive-work orchestrator. Consistency demands are what make it necessary — moving to a new CUDA version, for instance, may require every relevant host on a new driver first. With thousands of hosts and overlapping planned and unplanned operations, OpsPlanner serializes overlapping scopes, takes hosts safely out of and back into production, verifies upgrades before a host returns, and carries a built-in handover flow with correct escalation that avoids overlaps and deadlocks. It also owns and safeguards the planned-maintenance and failure buffers. The scale claim: a million operations a day.
The safety net underneath: trains auto-stop if maintenance or failure buffers run out, failing upgrades get offboarded automatically, and rollout phases ensure only well-tested changes reach global systems. When something still breaks, emergency trains and large-scale maintenance exist as fast paths.
My take
The reframe: this piece treats maintenance as capacity planning, not operations hygiene. The guarantee is a fraction of the fleet (“all minus one domain”), buffers are budgeted capacity with an owner, and domain size is chosen like an economic tradeoff between interruption cost and lost capacity. That’s a genuinely different mental model from “maintenance windows,” and it’s the part worth stealing.
The second insight follows from “bad hosts are very bad.” In a web-serving fleet, a slow machine sheds some traffic and life goes on; in synchronized training, a slow machine poisons the whole job. That inverts normal fleet intuition — host verification and firmware hygiene are performance engineering, not janitorial work.
The most copyable design is the consistency split: pin what the job ships, let the platform drift gradually beneath it. Any team running stateful workloads on a deep stack — databases on tuned kernels, anything with pinned driver versions — has the same problem in miniature.
Caveats: the article is qualitative. There are no numbers for domain sizes, SLOs, or interruption rates, so you can’t directly size your own program from it. And the 600,000-GPU figure was a plan at publication, not a built fact — read it as direction, not estate.
Where this is useful
- Running any synchronized distributed workload — training, simulation, HPC — where “maintenance window” has stopped being viable once jobs span many hosts.
- Capacity planning, by budgeting maintenance and failure buffers as first-class capacity with an explicit owner rather than borrowing from slack.
- Rollout design for deep platforms — the split between a consistent job stack and gradually drifting host stack generalizes well beyond AI.
- Overkill below a certain fleet size; a single cluster with tolerant users can keep whole-cluster windows and skip the machinery.
Apply this: For the most interruption-sensitive workload you run, write down what fraction of capacity you could remove at once without missing its deadline. Then check that this fraction is genuinely reserved as buffer — not quietly borrowed from headroom that a bad week will also want.
What I want to remember
Remember: Maintenance at scale isn’t a window you schedule; it’s capacity you reserve so jobs never notice. Keep the job’s own stack consistent, let the host’s stack drift gradually, and size each maintenance domain to what interruptions actually cost.