Authors: Siyuan Shen (ETH Zürich); Anton Korzh and John Bachan (NVIDIA Corporation); Tiancheng Chen (ETH Zürich); Arnav Goel, Ludwig Schneider, Pouya Kousha, Zhenhao He, Sylvain Jeaugey, Kamil Iskra, Nishank Chandawala, and Jeff Hammond (NVIDIA Corporation); and Torsten Hoefler (ETH Zürich)
Abstract: GPU collective communication is often optimized for bandwidth, yet many emerging workloads are latency-bound. Long-context LLM inference is a key example, where small collectives lie on the critical path of token generation, making microsecond-scale overheads impactful. In this work, we study how to approach the hardware speed-of-light (SoL) lower bound for GPU collectives within a scale-up network. We identify key design principles, including barrier-free synchronization and efficient use of symmetric memory and multicast. Building on NCCL’s device-side API, we develop low-latency interfaces for constructing custom kernels and implement new symmetric collectives in NCCL. Microbenchmarks show substantial latency reductions, achieving within 10% of the measured SoL bound. Integrated into applications, these kernels reduce inter-token latency and improve throughput in LLM inference, while also accelerating cuSOLVERMp, demonstrating benefits across AI and HPC workloads.
Presentation: file