Research and ACM SRC Posters Archive

Understanding and Improving Multi-Rack Symmetric-Memory AllGather in NCCL


Poster Type: Research Posters

Author: Prajwal Singhania (University of Maryland, NVIDIA Corporation), Vijay Korthikanti (NVIDIA Corporation), Jared Casper (NVIDIA Corporation), Mohammad Shoeybi (NVIDIA Corporation), Abhinav Bhatele (University of Maryland)

Abstract: Rack-scale GPU systems such as the GB200 NVL72 extend the NVLink scale-up domain from four or eight GPUs per server to 72 GPUs per rack. Large language model (LLM) training still scales across many racks, with all-gathers spanning heterogeneous NVLink and InfiniBand fabrics. These all-gathers operate in a message-size regime where both latency and bandwidth matter. In this work, we characterize NCCL's symmetric-memory and legacy all-gather performance for this regime. We derive a closed-form topology-aware bandwidth bound and extend it with a latency term for comparison with empirical measurements. Based on observed trends and model insights, we design two new GPU-initiated symmetric-memory all-gather implementations that outperform NCCL's symmetric-memory all-gathers by up to 2.6x for 32-512MB messages.

Best Poster Finalist (BP): no
Poster: PDF
Poster Summary: PDF


Back to Poster Archive Listing