Workshop: 17th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Heterogeneous Systems (ScalAH'26)
Authors: Amirreza Barati Sedeh, Ryan Eric Grant, and Ahmad Afsahi (Queen's University)
Abstract: Broadcast, Scatter, and Gather are MPI collectives critical to HPC and DL application scalability. As GPU clusters rely on multiple NICs per node for inter-node bandwidth, traditional single-tree algorithms fail to exploit this bandwidth for large messages. We propose a GPU-based, multi-NIC-aware design for large-message MPI Scatter, Gather, and Broadcast that partitions each collective across multiple independent binomial trees, each mapped to a distinct NIC, enabling communication to proceed concurrently across all rails. We implement and evaluate our design in Open MPI on two GPU platforms, full-mesh NVLink and NVSwitch, at 16, 32, and 64 GPUs. Compared to Open MPI, our design achieves latency reductions of 52.5%, 82.8%, and 74.6% for Scatter, Gather, and Broadcast on full-mesh, and 57.0%, 79.8%, and 74.3% on NVSwitch. A step-complexity analysis shows our design reduces critical-path communication rounds from ⌈log2 p⌉ to ⌈log2(p/g)⌉, where g is rails per node, yielding better scalability at scale.
Back to 17th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Heterogeneous Systems (ScalAH'26) Archive Listing
Back to Full Workshop Archive Listing