Workshops Archive

Evaluating In-Network Collectives for AI/ML Workloads with ATLAHS


Workshop: Workshop on High Performance Fabrics for AI and HPC

Authors: Wanja Stämpfli, Mikhail Khalilov, Tommaso Bonato, and Timo Schneider (ETH Zürich); Cristian Fraschetti and Daniele De Sensi (Sapienza University of Rome); and Torsten Hoefler (ETH Zürich)

Abstract: In-network collectives (INC) can accelerate many collective network operations by moving computations away from the CPU, either into the NIC or into switches. The benefits are reductions in both traffic and latency, at the cost of higher system complexity. Designers of AI/ML training and inference systems have to balance network costs and the performance impact of network features on current and potentially future workloads. The ATLAHS toolchain allows us to study the impact of latency, bandwidth, and topology changes on workload performance, while supporting simulations of unprecedented scale. In this work, we add INC support to ATLAHS and report the impact on collective performance for groups of up to 1024 ranks, and on Llama-2-7B training performance at up to 64 GPUs.


Back to Workshop on High Performance Fabrics for AI and HPC Archive Listing

Back to Full Workshop Archive Listing