Workshop: ExaMPI26: Workshop on Extreme-Scale MPI
Authors: Varun Chotalia (Stony Brook University); Joseph Schuchart (Stony Brook University, Institute for Advanced Computational Science (IACS)); and Jeff Hammond (NVIDIA)
Abstract: CUDA interprocess communication (IPC) allows direct pointer access to GPU allocations of peer processes. Applications are required to handle export, collective exchange, per-peer import, pointer caching, failure handling, and cleanup. We show that these operations can be managed through an MPI window instead, using existing MPI shared-window query mechanisms. We implement this approach in WinIPC, a library that intercepts MPI window operations through the MPI profiling interface. We evaluate the approach using distributed matrix transpose, five-point stencil, and CUDA-enabled LULESH. We demonstrate peer-pointer access across 32 GPUs on eight physical nodes of a GB200 NVL system. For distributed transpose with accumulation at matrix order 1024, the single-kernel direct implementation achieves 2.44x the throughput of GPU-aware MPI. At matrix order 6912 on 32 GB200 GPUs, this variant reaches 20x the GPU-aware MPI rate. Moreover, we identify an allocation-time restriction of fabric-enabled IPC that limits its use with container-managed GPU buffers.
Back to ExaMPI26: Workshop on Extreme-Scale MPI Archive Listing
Back to Full Workshop Archive Listing