Technical Papers Archive

Scalable and Efficient Simulation of LLM Training


Authors: Yicheng FENG, Kin Hang Sew, Yuetao Chen, and Kaiwen Chen (CUHK); Jingzong Li (The Hang Seng University of Hong Kong); Tianyuan Wu (Hong Kong University of Science and Technology, Guangzhou); Zheng Zhou (Sun Yat-sen University); Peng Cheng (Microsoft Corporation); Chuan Wu (The University of Hong Kong); Wei Wang (Hong Kong University of Science and Technology, Guangzhou); and Tsung-Yi Ho and Hong Xu (CUHK)

Abstract: Simulation offers unique values for both enumeration and extrapolation purposes, and is becoming increasingly important for managing the massive machine learning (ML) clusters and large-scale distributed training jobs. In this paper, we build Echo to tackle three key challenges in large-scale training simulation: (1) tracing the runtime training workloads at each device in an ex-situ fashion so we can use a single device to obtain the actual execution graphs of 1K-GPU training; (2) accurately estimating the collective communication without high overheads of discrete-event-based network simulation; and (3) accounting for the interference-induced computation slowdown from overlapping communication and computation kernels on the same device. Echo supports both dense and MoE workloads with high fidelity and further enables GPU memory-aware ex-situ simulation. Echo delivers on average 8% error in training step (~3x lower than state-of-the-art simulators) for GPT-175B on a 256-GPU H800 cluster with 3D parallelism on Megatron-LM under three minutes.


Presentation: file

Back to Technical Papers Archive Listing