Workshop: FTXS: Workshop for Faults, Trustworthiness, and eXplainability for AI Systems at Scale
Authors: Elisabeth Giem (Sandia National Laboratories; University of California, Riverside); Saibal De, Hemanth Kolla, and Jackson Mayo (Sandia National Laboratories); Martin Schatz (Meta Platforms, Inc.; Sandia National Laboratories); Jeremy Myers (Qualcomm Technologies, Sandia National Laboratories); and Zizhong Chen (University of California, Riverside)
Abstract: Distributed memory linear solvers are the workhorse of many scientific HPC applications and are also close counterparts to AI algorithms. The increasing scale and heterogeneity of HPC systems renders them susceptible to silent data corruption (SDC). Physics based checksums (PBCs) can detect SDC errors locally on each computing node. Detection can be followed by rollback to a checkpoint: however, this localized checkpointing does not scale optimally as fault rates increase. Increased asynchrony in Krylov solvers promotes scalability of fault recovery from SDC errors. We focus on the s-step communication-avoiding conjugate gradient (CACG), and describe the algorithmic changes needed to implement PBCs to protect all portions of the CACG algorithm. Our results show that local recovery incurs much lower overhead and remains scalable at much higher node counts and fault rates compared to global recovery and that as fault rates increase, the benefit of asynchrony increases.
Back to FTXS: Workshop for Faults, Trustworthiness, and eXplainability for AI Systems at Scale Archive Listing
Back to Full Workshop Archive Listing