Authors: Kenneth Sebastian, Devendra Ayalasomayajula, Annu Jolly, Rashi Motwani, Abby Kuang, Giorgio Di Guardia, Aman Miryala, Sivakumar Arayandi Thottakara, Ranjitha Prasanna, and Satvir Kooner (NVIDIA Corporation)
Abstract: Reliability is a critical challenge in operating large-scale GPU training infrastructure across multiple cloud providers. Despite growing interest in GPU fault characterization, no prior work describes the automated systems that manage faults across providers at scale, nor a methodology for continuous reliability improvement. We operate a large GPU training fleet spanning all major CSPs and NCPs, encompassing various GPU architectures with close to 1 billion GPU hours/year. We focus on three production clusters at the highest maturity level (≥98% node availability) across three providers, analyzing 19,564 fault incidents across 5,065 nodes over 5–7 months. To maintain availability, we built a breakfix system with three pillars: scale-up automation, fault triage with provider-specific remediation actions, and repeat-offender management. We report on reliability KPIs: mean time to repair (MTTR), automated triage success rate (ATSR), node recidivism rate (NRR), and present a Pareto-driven methodology for continuous improvement.
Presentation: file