Meta opens up Prometheus: a gigawatt-scale supercluster that trained Muse, heading for roughly 1 million GPUs and 99% fewer interruptions

Start with the basics. On October 19, Meta Engineering published 'Inside Prometheus: Meta's Journey to a Gigawatt-Scale Supercluster', by Kevin Lee, Yashar Bayani, Ahmed Sharif and Yee Jiun Song. The site is New Albany, Ohio: two years ago a conventional data center running web traffic, now reworked to train frontier models. The build is phased, phase one is complete, and it was used to train the Muse family of foundation models — including Muse Wonder, which Meta calls a new frontier model for personal and professional agentic tasks. Full scale is roughly 1 million GPUs; today it comprises hundreds of thousands across multiple interconnected buildings on one campus, functioning, in Meta's words, as a single machine.

The first metric that actually matters here is goodput, not peak floating-point throughput. Meta says that by improving stability and reliability it achieved more than 85% goodput for Muse Wonder pre-training and over 90% for reinforcement learning runs, with an overall wall-clock mean time between interruptions of roughly 47k GPU hours during those runs; subsequent fleet-wide firmware fixes pushed that past 54k GPU hours and work continues. Why it matters: in a cluster spanning multiple buildings and a hundred-plus kilometres of fibre, a fault on a single host can slow or stop an entire training job, and when goodput drops you have simply bought idle silicon.

The hardware is two generations deep with a third on the way. Catalina, the first platform, is built on NVIDIA GB200 NVL72 in the Ariel variant — a two-rack configuration where 36 GPUs per rack are linked by NVLink into a single 72-GPU domain. One detail matters: Ariel pairs each Blackwell GPU with a Grace CPU and two ConnectX-7 400G NICs at a 1:1 CPU-to-GPU ratio, which let Meta deploy 800G of scale-out per GPU earlier than the rest of the industry. Meta explains the motive: GB200 NVL72 scaled up nearly every other dimension versus H100 — FLOPS, HBM bandwidth, scale-up bandwidth, domain size — while scale-out bandwidth per GPU would have stayed at 400G, and that imbalance was expected to cost performance. Per domain: 72 GPUs, 13.4 TB of HBM3e (192 GB each), 720 peak PFLOPS FP8 (1,440 at FP4), 130 TB/s NVLink, 576 TB/s aggregate memory bandwidth, and about 120 kW across the two-rack pair.

The second platform, Clemente, moves to NVIDIA GB300 NVL72 and changes both the GPU and the platform. On the GPU side HBM rises from 192 GB to 288 GB per device (1.5x), dense FP4 throughput improves by a similar factor, and attention (softmax) acceleration roughly doubles. On the platform side the shift is bigger: ConnectX-8 SuperNICs deliver 800G per GPU natively, so what Catalina deployed early becomes baseline, and with a 2:1 GPU-to-CPU ratio the full 72-GPU domain fits in a single rack, greatly simplifying physical deployment. Meta's specs: 72 Blackwell Ultra GPUs, 20.7 TB of HBM3e, over 1 exaFLOP of peak FP4, 130 TB/s NVLink (1.8 TB/s bidirectional per GPU), about 135 kW per rack, and 5-10% better performance than GB200 NVL72. Cooling can be modular Air-Assisted Liquid Cooling or facility liquid cooling in the new data centers, with less cabling. The third generation will be based on NVIDIA Vera Rubin NVL72, described as arriving soon.

Networking is, I think, the most underrated part of this post. Building-level fabrics are Ethernet and interconnect up to 20,736 GPUs each. Early on Meta used a Disaggregated Scheduled Fabric to handle training collectives — a few large, long-lived flows that collide under conventional load balancing — via sub-packet multipath routing and packet reordering in the fabric, with virtual output queues and credit-based end-to-end flow control. As Ethernet NICs gained endpoint packet reordering, Meta moved to a Non-Scheduled Fabric using shallow-buffer 51.2T switches such as Minipack3 with dynamic load balancing, and Minipack3N based on NVIDIA's Spectrum-4 ASIC with adaptive routing on ConnectX NICs. Both run on Meta's own FBOSS stack.

Cross-building is harder still. The Backend Aggregation network is a multi-layer hub-and-spoke fabric connecting building fabrics, spanning 2 km to over 100 km, built on Arista 7800-series chassis with deep-buffered Broadcom Jericho3 silicon and delivering multiple Pbps per region pair. Running pre-training across those distances required dedicated co-optimisation with Meta Superintelligence Labs, Meta says, because the speed-of-light constraint alone would otherwise slow training. The whole network adds up to millions of miles of fibre strand, tens of thousands of switches and millions of optics — 2x400G FR4 LITE short-reach, low-cost pluggables inside buildings, and 800G-ZR coherent optics beyond roughly 10 km. This stopped being data-center networking and became an inter-regional computer interconnect.

Then the arithmetic behind that 99%. Meta defines an interruption as any fault that forces a job to pause and enter a recovery cycle — GPU silicon defects, network fabric anomalies, storage bottlenecks, thermal limits. In phase one in mid-2025 interruption rates peaked near 1,000 events per 8,000 GPUs per day; a year of cross-functional engineering cut that by 99%, enough to sustain multiple training jobs of tens of thousands of GPUs in parallel. Decomposed, there are four categories. NVLink signal integrity was the main bottleneck of early GB200 deployment: with over 600,000 NVLink connections, marginal degradation triggers recovery cycles, so Meta added continuous telemetry and automated flagging plus pre-production L11 diagnostic burns to isolate weak links. Memory reliability (HBM and LPDDR) became a major source as GB300 scaled capacity, addressed with HBM firmware settings, memory scrubbing, an improved LPDDR error-correction algorithm and pattern-based predictive detection that retires degrading devices before they hit throughput. NVLink switch and fabric health is monitored at millisecond resolution because one switch fault can compromise an entire NVL72 domain, with automated recovery paths and traffic rebalancing. Backend network events at the BAG super-spine layer are handled by hardware fault detection, with planar topologies rerouting around failed fibre or ports.

Fault recovery uses two mechanisms that fit together neatly. PAFT (parallelism-aware fault tolerance, arXiv 2602.00277) handles faults isolable to a single data-parallel replica: that replica goes offline, the remaining cluster repeats the step and keeps training productively, and the compromised slice restarts on spare hosts and resynchronises, converting a total halt into a temporary dip in aggregate throughput. Checkpointing covers faults whose blast radius exceeds a replica, restarting the whole job from the last checkpoint; Meta's distributed checkpointing asynchronously saves model state including gradients and optimizer data to multi-tier storage, with delta checkpointing and in-memory staging to keep the throughput cost small. Both combine with a sparing strategy using pre-allocated spare GPU trays inside racks plus spare racks to shorten time-to-replacement.

There is also an operational layer built for scale that reads more like manufacturing than internet infrastructure. GB200 NVL72 expanded the scale-up domain from eight GPUs in the H100 era to 72, increasing the fault blast radius ninefold, so hardware maturation became a discipline in its own right: firmware and driver bundles deploy to more than 100k GPUs within hours, and P90 single-pod upgrade time fell from over seven hours early on to two hours; automated signal screening validates more than 600,000 NVLink paths before production traffic; power delivery is stress-tested under the 1,400W load of individual GPUs to surface failure modes that only appear at volume; strict quality gates on cold plate and manifold assembly guard against leaks; and health checks, prepositioned spares and an attribution-driven accountability framework decompose every halt by root cause, with Meta owning scheduling and infrastructure and NVIDIA owning firmware, signal diagnostics and memory handling. Meta says that attribution removed ambiguity and sped up fixes on both sides.

Boundaries, stated plainly. First, this is an engineering blog post, not a financial filing or an independent benchmark; every goodput, MTBI and interruption-rate figure is Meta's own claim and unverified externally. Second, roughly 1 million GPUs is a full-scale target — Meta explicitly says it runs hundreds of thousands today, so do not read the target as the present. Third, that Muse Wonder trained on these clusters is Meta's statement, but how many GPUs, for how long and at what cost is not disclosed. Fourth, the third-generation Vera Rubin NVL72 platform is described as arriving soon with no timetable, and this article does not invent one. Fifth, the drop from roughly 1,000 interruptions to a tenth of that is operational data from Meta's own fleet and does not transfer to other operators' clusters.