Technical Papers Archive

Evaluation and Optimization of Communication Compression for 10K-GPU LLM Training


Authors: Xingchen Liu, Daran Sun, Xingjian Tian, and Man Liu (Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS); University of Chinese Academy of Sciences); Haoran Kong (Shenzhen Loop Area Institute; The Chinese University of Hong Kong, Shenzhen); Jingwu Yang, Shengkai Lyu, and Yiqin Cui (Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS); University of Chinese Academy of Sciences); Shengquan Yin (University of Science and Technology of China); Zheng Wei, Yida Gu, Wenjing Huang, Bing Lu, Zedong Liu, Hairui Zhao, and Zhenlong Ma (Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS); University of Chinese Academy of Sciences); Xiaosen Shan and Juan Pang (Dawning Information Industry Co., Ltd.‌); Fan Yang and Zhan Wang (Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS); University of Chinese Academy of Sciences); Zizhong Chen (Shenzhen Loop Area Institute; The Chinese University of Hong Kong, Shenzhen); and Dingwen Tao and Guangming Tan (Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS); University of Chinese Academy of Sciences)

Abstract: Collective communication is critical for scaling large language model (LLM) training to megascale GPU clusters. However, communication becomes a major bottleneck limiting scalability. Existing approaches improve bandwidth utilization but are constrained by the physical network in large-scale clusters. Communication compression reduces data volume and offers a way to break the communication wall, yet its performance and accuracy at extreme scale remain unclear. To bridge this gap, we present the first systematic study of communication compression across scales up to 10K GPUs, showing that it maintains accuracy but suffers from limited performance gains due to bandwidth hierarchy limitations, weak communication scaling, and limited resource awareness. We enhance the co-design of communication and compression with a hierarchical method that restructures communication compression, adapts communication algorithms to each scale, and tunes network resource allocation. Our approach achieves up to 5.87x communication improvement and 1.73x end-to-end training speedup on Llama3 models at 10,240 GPUs.


Presentation: file

Back to Technical Papers Archive Listing