Workshops Archive

LinkPass: Transparent RDMA Link Failover and Recovery for Collective Communication


Workshop: Workshop on High Performance Fabrics for AI and HPC

Authors: Akram Sbaih (Clockwork Systems); Katie Gioioso (NVIDIA Corporation); Rahul Bothra (University of Illinois Urbana-Champaign); Yi-Ting Wu, Vinay Sriram, and Yilong Geng (Clockwork Systems); and Balaji Prabhakar (Stanford University, Clockwork Systems)

Abstract: Optical links in high-bandwidth AI clusters are prone to failures from temperature, vibration, dust, and other factors, causing temporary or permanent disconnections or intermittent packet loss. In tightly synchronized distributed training, a single flapping link between a network interface card (NIC) and top-of-rack switch can stall every participating rank. We present LinkPass, a shim beneath the network-plugin interface shared by NVIDIA NCCL and AMD RCCL that transparently masks such failures. LinkPass uses an RDMA probe mesh to detect stalling links, fails traffic over across healthy NUMA-proximate NICs to avoid backup bottlenecks, and reverts traffic when the original NIC recovers. Operating entirely over standard InfiniBand Verbs, LinkPass is transparent to collective libraries and training code and agnostic to GPU and NIC vendors. Evaluations show that LinkPass prevents link flaps from crashing jobs, significantly increasing mean time between failures and reducing required checkpointing frequency.


Back to Workshop on High Performance Fabrics for AI and HPC Archive Listing

Back to Full Workshop Archive Listing