Authors: Mingshu Zhai, Minkun Guo, Hongrui Lian, Shuangyu Li, and Qi Zhang (Tsinghua University); Chen Hu (Unaffiliated); Xiawei Du and Yuyang Jin (Tsinghua University); Zan Zong (University of Science and Technology Beijing); and Jidong Zhai (Tsinghua University)
Abstract: Fine-grained Mixture-of-Experts (MoE) is the de facto architecture for scaling large language models. However, dynamic expert load imbalance causes severe memory fluctuations, degrading training efficiency. Existing systems rely on static memory provisioning: they either over-provision, leaving GPU memory idle during normal execution, or under-provision, inviting out-of-memory crashes and costly manual rollbacks when load spikes occur.
To address these challenges, we propose BaGuaLu-2, a large-scale MoE training system that eliminates this dilemma through flexible provisioning. By transforming memory allocation from a static, one-time tuning choice into a continuous and fine-grained control, BaGuaLu-2 grants exact activation memory on demand for each training batch. Furthermore, it offsets any dynamic management overhead through system optimizations. BaGuaLu-2 demonstrates exceptional scalability, training a model with 1,040 billion parameters across 12,288 GPUs while sustaining over 2 EFlops at 35% MFU. Compared with state-of-the-art systems, BaGuaLu-2 delivers 1.14× and 1.13× average throughput improvements over Megatron-Kwai and AdaPipe, respectively.
Presentation: file