Poster Type: Research Posters
Author: Emir Gencer (University of Maryland), Abhinav Bhatele (University of Maryland), Sajal Dash (Oak Ridge National Laboratory (ORNL))
Abstract: Mixture-of-Experts (MoE) training workloads use imbalanced AlltoAllv collectives at bandwidth-bound message sizes. On Frontier, one-sided AlltoAll can beat RCCL and Cray-MPICH, but only with the right schedule and only as per-peer fetches shrink. Other approaches balance load across a node's NICs by explicitly copying data between GPUs, which costs GPU staging buffers and overlaps that copy with network traffic only coarsely. We introduce CURLEW, a completion-driven, GPU-staging-free RDMA AlltoAllv with dynamic local-NIC balancing. CURLEW achieves near-flat bandwidth across GPU counts and message sizes on balanced patterns, is 1.9x faster than previous work at max-to-mean skew above 8, and reduces end-to-end training time of a 201B-parameter model by 22% at a capacity factor of 1.25.
Best Poster Finalist (BP): yes
Poster: PDF
Poster Summary: PDF