Workshops Archive

Co-designing MoE Architectures and System Stacks for Scalable Training on Leadership Supercomputers


Workshop: AI on HPC: Performance Engineering, Challenges and Opportunities

Authors: Sajal Dash (Oak Ridge National Laboratory), Yueming Yuan (University of Illinois Urbana-Champaign), Junqi Yin (Oak Ridge National Laboratory), Minjia Zhang (University of Illinois at Urbana-Champaign), and Feiyi Wang (Oak Ridge National Laboratory)

Abstract: Large language models (LLMs) have shown exceptional promise as scientific foundation models. However, substantial pre-training costs pose a major bottleneck in training high-quality models. The Mixture-of-Experts (MoE) has emerged as an effective solution to this problem. Preceding MoE studies have been conducted on NVIDIA platforms, while many scientific foundation models are trained on supercomputers with diverse accelerators. By examining pretraining strategies on a dataset sourced from scientific texts and porting state-of-the-art MoE training techniques to the ROCM-stack on Frontier, we identify crucial factors to achieve high training throughput. For the first time, we show that it is possible to scale training of massive MoE models on clusters with thousands of GPUs. Leveraging these insights, we train a suite of MoE models for science, called SMOREs. Evaluations of these models against standard benchmarks such as SciQ and ARC-Challenge show that they outperform their dense counterparts in the early phases of training.


Back to AI on HPC: Performance Engineering, Challenges and Opportunities Archive Listing

Back to Full Workshop Archive Listing