Workshop: ExaMPI26: Workshop on Extreme-Scale MPI
Authors: Amirreza Barati Sedeh, Ryan Eric Grant, and Ahmad Afsahi (Queen's University)
Abstract: Alltoall is a communication-intensive collective, realizing a full-mesh exchange among all processes, which relies on RDMA's Reliable Connection (RC) transport for inter-node communication, requiring a dedicated Queue Pair (QP) per peer and exhausting on-NIC memory as peer count grows. We propose a GPU-aware MPI Alltoall that avoids vendor-specific transports, decomposing the full-mesh exchange graph into edge-disjoint Hamiltonian cycles so each process has a small, fixed number of direct peers, combined with a pipelined, multi-NIC exchange overlapping intra- and inter-node communication. Experimentally, at 36 and 72 GPUs, we reduce measured QPs per NIC by up to 98% and 93% versus Open MPI and NCCL, matching an analytical model projecting 66%-98% reductions up to 1152 GPUs. Our design achieves up to 34% and 36% lower Alltoall latency than Open MPI and MVAPICH-Plus, matches NCCL's latency using MPI, and delivers up to 1.2x and 1.66x higher heFFTe throughput than Open MPI and MVAPICH-Plus.
Back to ExaMPI26: Workshop on Extreme-Scale MPI Archive Listing
Back to Full Workshop Archive Listing