Workshop: The 9th Annual Parallel Applications Workshop, Alternatives to MPI+X (PAW-ATM)
Authors: Aaron Benjamin Haarti and Kari Rummukainen (University of Helsinki)
Abstract: Overlapping communication with computation is necessary at scale, and the communication library sets how cheaply that overlap can be had. We study point-to-point halo exchange overlapped with compute on a GH200 Grace Hopper platform with InfiniBand, comparing host-initiated GPU-aware MPI against device-initiated NVSHMEM and NCCL. Using a synthetic benchmark, we determine for which shared resource each backend competes. We then apply the communication models to a memory-bound workload, the SU(3) staplesum implemented with the hila framework. We find that in the memory-bound regime the constraint is HBM utilization rather than SM occupancy with an added L2 cost consistent with evicting the compute kernel's reused working set. Device-initiated NVSHMEM is fastest wherever communication is exposed, reaching 99% overlap efficiency in weak scaling at 128 GPUs and extending usable strong scaling 4X from 16 to 64 GPUs over non-overlapped execution. Every optimization reduces either L2 pressure or latency overhead in point-to-point communication.
Back to The 9th Annual Parallel Applications Workshop, Alternatives to MPI+X (PAW-ATM) Archive Listing
Back to Full Workshop Archive Listing