Technical Papers Archive

OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling


Authors: Jihu Guo and Tenghui Ma (Fudan University, Shanghai AI Laboratory); Wei Gao (Hong Kong University of Science and Technology, Guangzhou); Dahua Lin (The Chinese University of Hong Kong, Shenzhen; Sensetime Research); Peng Sun, Xun Chen, and Jiaxing Li (Independent Researcher); Yuyang Jin (Tsinghua University); and Zhisheng Ye (Independent Researcher)

Abstract: Pipeline parallelism is widely used to train large language models (LLMs). However, increasing heterogeneity in model architectures exacerbates pipeline bubbles. Prior approaches typically optimize a single phase of the pipeline schedule (i.e., partitioning, placement, or scheduling), leaving substantial bubbles. While promising, co-optimization poses three key challenges: (1) complex performance modeling, (2) a combinatorial search space, and (3) irregular execution orders.

We propose OctoPipe, a pipeline parallelism system to co-optimize partitioning, placement, and scheduling. First, we build a graph-based pipeline simulator to model heterogeneous pipeline execution for co-optimization. Second, we develop an iterative bubble-aware tuner to efficiently explore the combinatorial search space. Third, we implement a unified pipeline executor that dynamically orchestrates computation and communication to support irregular execution orders without deadlocks while maximizing communication-computation overlap.

Experiments show that OctoPipe achieves 1.09x-1.49x throughput improvement over the state-of-the-art pipeline parallelism approaches across various heterogeneous model configurations and GPU cluster scales.



Presentation: file

Back to Technical Papers Archive Listing