Authors: Zixian Wang and Liheng Jing (University of Illinois Urbana-Champaign); Emily Herron, Sajal Dash, and Feiyi Wang (Oak Ridge National Laboratory (ORNL)); and Minjia Zhang (University of Illinois Urbana-Champaign)
Abstract: We present X-MoE-4D, a distributed training system for scaling expert-specialized Mixture-of-Experts models on hierarchical HPC platforms. We observe that existing expert-parallel approaches suffer from severe scalability limits due to cross-node all-to-all communication. To address this, X-MoE-4D adopts a pipeline-global, expert-local execution model that confines expert computation within nodes while scaling across nodes via pipeline parallelism. X-MoE-4D further introduces a Minimax Pipeline Planner that jointly optimizes pipeline partitioning and activation checkpointing to minimize bottleneck stage execution time under memory constraints. Together, these techniques eliminate cross-node expert routing and mitigate MoE pipeline skew. We evaluate X-MoE-4D on the Frontier supercomputer, demonstrating the first efficient one-trillion-parameter expert-specialized MoE training on an AMD-based HPC platform. X-MoE-4D achieves 2.24× higher aggregate throughput than X-MoE at one-trillion-parameter scale (2.4× per GPU throughput) while using fewer GPUs (960 vs. 1,024), and sustains a 3.24× speedup when strong-scaling the 173B model to 2,304 GPUs. Code available at https://github.com/Supercomputing-System-AI-Lab/X-MoE.
Presentation: file