Tens Of Thousands Of Nvidia GPUs Sit Idle In Meta's Prometheus Supercluster, Waiting For Failure
"Arthur T Knackerbracket" writes:
More than 11 percent of the GPUs at Meta's Prometheus supercluster in New Albany, Ohio, are either dead or sitting idle waiting to step in when the next drops out, Zuckercorp revealed this week.
The massive pool of redundant GPUs is just one of several measures the hyperscaler has taken to keep its AI training runs on track.
Prometheus is among Meta's largest compute deployments to date. Powered by Nvidia's Grace Blackwell and (soon) Vera Rubin rack systems, Meta says the supercluster will eventually scale to a gigawatt of capacity spanning about a million accelerators.
But as anyone who has worked on a large scale compute cluster understands, whether for LLM training or traditional HPC workloads, more chips inevitably means more problems.
In an eight-GPU node, the chance of having a GPU fail is relatively low. In a 100,000-GPU cluster, failures are unavoidable. Meta is building something that's 10x larger and requires distributed training runs across multiple datacenters. Keeping the entire thing humming long enough to reach the next checkpoint becomes a major headache.
In a blog post published this week, Meta offered a unique look at the work it's done to keep failures from derailing the development of models like Muse Spark or its most recent release Muse Wonder.
When Meta began the first phase of its Prometheus deployment in New Albany, the company says, it was experiencing roughly 1,000 interruptions per day for every 8,000 GPUs.
Interruptions don't mean GPU failure. They simply mean something crashed, failed, or timed out, forcing the training run to stop and roll back to the last known good checkpoint.
Meta claims in the year since it's been able to cut down the number of interruptions by 99 percent, to about 11 per day per 8,000 GPUs.
One of the ways it's done this is by keeping some of Prometheus' GPUs in reserve. The cluster is currently powered by a combination of GB200 and GB300 systems, each with 72 GPUs per NVLink domain. However, only 64 of them are ever in use at any given moment. The other eight are essentially hot spares.
Meta did this because a single failed GPU would take out an entire rack and interrupt the training run. Now, when a GPU fails, one of the spares can slot in and take its place. While this saves time - Meta doesn't have to drop the entire rack from the cluster or wait for someone to swap the faulty component to start training again - a failed GPU would still result in an interruption, albeit a shorter one.
To mitigate this, Meta took advantage of the fact that large scale training jobs usually use data parallelism, where multiple copies of the model are resident within the cluster.
Read more of this story at SoylentNews.