Research and ACM SRC Posters Archive

Understanding AllReduce Performance Through Hardware Telemetry


Poster Type: Research Posters

Author: Ram Sharan Chaulagain (Argonne National Laboratory), Nathan Nichols (Argonne National Laboratory), Servesh Muralidharan (Argonne National Laboratory), Kalyan Kumaran (Argonne National Laboratory)

Abstract: Inter-GPU AllReduce operations traverse several tightly coupled hardware components, including the GPU, PCIe interface, NIC, and network. Each component exposes hardware telemetry for its local behavior, and no single telemetry source observes the complete communication path. A combined view is therefore needed to narrow where performance is limited and to guide optimization. We combine selected NVIDIA A100 GPU counters with Cassini NIC counters on HPE Slingshot-11 to analyze a 1 GiB inter-node AllReduce across NCCL protocols, NCCL channel counts, and GPU-aware MPI versus NCCL. Across the three studies, the joint telemetry associates performance differences with distinct behaviors: work amplification for NCCL LL, increased synchronization and reduced transfer progress at high channel counts, and slow transfer progress with additional rank imbalance for MPI. Joint GPU–NIC telemetry therefore helps distinguish different sources of AllReduce slowdown and narrows where optimization effort should be focused.

Best Poster Finalist (BP): no
Poster: PDF
Poster Summary: PDF


Back to Poster Archive Listing