Birds of a Feather Archive

Communication Libraries in HPC and AI


Authors: Nikoli Dryden (Lawrence Livermore National Laboratory (LLNL)), Shelby Lockhart (Advanced Micro Devices (AMD)), Didem Unat (Koç University, Turkey), Jeff Hammond (NVIDIA Corporation)

Abstract: The world's largest supercomputers for scientific discovery are also premier systems for artificial intelligence model training and inference. While traditional HPC compute has predominately leveraged the MPI standard, AI workloads have increasingly focused on collective communication libraries, such as NVIDIA's NCCL and AMD's RCCL, which are optimized for high throughput and GPU-native usage. This Birds of a Feather (BoF) session at the SC Conference (SC) aims to delve into the intricacies of collective communication libraries, focusing on the comparison between the widely adopted Message Passing Interface (MPI) and NCCL/RCCL, as well as other key messaging libraries, such as SHMEM.

Long Description: The world's largest supercomputers for scientific discovery are also premier systems for AI model training and inference. However, while traditional HPC compute has predominately leveraged MPI, AI workloads have increasingly focused on collective communication libraries such as NVIDIA's NCCL, AMD's RCCL, and Intel’s OneCCL (*CCLs), which are optimized for high-throughput and GPU-native usage. With increasing emphasis on AI inference, use of device-side, latency-optimized communication has also increased.

As a continuation of a well-attended BoF at SC25, this BoF continues the exploration of collective communication libraries and their role in modern HPC and AI systems. The session will focus on the strengths, limitations, and interoperability of MPI, *CCLs, and other communication frameworks (e.g., *SHMEM).

While MPI has been the cornerstone of HPC communication for decades, GPU-native communication presents opportunities for performance enhancement through techniques like kernel fusion (increasingly important in latency-bound regimes in both AI and HPC). These libraries leverage hardware-specific optimizations to accelerate communication, particularly in GPU-centric environments, such as fine-grained communication/computation interleaving (e.g., with device-side *SHMEM libraries).

The session will cover the following key topics: * Overview of Collective Communication Libraries: Introduction to MPI/*CCLs/etc., highlighting core functionalities and potential uses in AI and HPC. * Performance Comparison: Analyzing the performance benefits of *CCLs, such as kernel fusion, and how they compare to MPI in various scenarios. * Case Studies and Real-World Applications: Discussions from experiences on leadership computing systems where collective libraries and large scale AI training runs have been performed. * DevTools: Are debuggers and profilers developed for traditional MPI-based applications still useful for AI applications using *CCLs? What new technologies should the community design to ensure productive developers on AI and HPC workloads? * Future Directions: Exploring the future of communication libraries, whether they are converging or diverging from standards. This will also discuss how libraries, along with the MPI standard, might be extended.

The session is designed to be highly interactive, encouraging participants to share their experiences, challenges, and insights.

Expected outcomes: * Enhanced awareness within the HPC community of the capabilities and limitations of MPI vs. AI-focused *CCLs. * Increase awareness of existing tools designed for AI and their adoption in HPC. * Insights into performance optimization techniques specific to GPU-based systems.

This session is relevant to the HPC community in general but specifically: * System architects and admins who configure and maintain supercomputers. * Communication and numerical library developers. * Application developers who are exploring the integration of complex workflows that leverage AI with HPC on supercomputers. * Vendors who are designing the next generation of network fabrics.

An open dialogue on the state of the community around communication libraries, along with their benefits and challenges, will educate the SC user community and also provide a lens by which to evaluate future performance goals, software requirements and standards, supporting tools and candidate system architectures. As exascale computing transitions into the era of large-scale AI and AI-for-Science workloads, a variety of communication libraries are expected to remain important and prevalent. It is important for application developers and system architects to understand, and discuss, the tradeoffs among them.


Back to Birds of a Feather Archive Listing