Technical Papers Archive

FluidGPU: Fine-Grained Kernel Disaggregation for Large Model Inference on Heterogeneous GPUs


Authors: Tiancheng Hu (Peking University); Jin Qin and Lei Chen (Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS); University of Chinese Academy of Sciences); Junhao Hu and Yuzheng Wang (Peking University); Mingxing Zhang (Tsinghua University); Chunwei Xia (University of Leeds); Yizhou Shan (Independent Researcher); Huimin Cui (Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS); University of Chinese Academy of Sciences); Ting Cao (Tsinghua University); Zheng Wang (University of Leeds); Xie Tao (Peking University); and Chenxi Wang (Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS); University of Chinese Academy of Sciences)

Abstract: This paper presents FluidGPU, the first kernel disaggregation system to improve performance and cost efficiency on heterogeneous GPUs for large model inference. Our key insight is that kernels within a single application exhibit diverse resource demands, making them the most suitable granularity for aligning computation with hardware capabilities. FluidGPU integrates offline analysis with online adaptation by extracting precise inter-kernel dependencies from PTX to ensure correctness, overlapping communication with computation through a pipelined execution model, and employing workload-aware scheduling with lightweight runtime adaptation. Extensive evaluations across five heterogeneous GPUs and four model architectures, scaling up to 16 GPUs, show that FluidGPU improves serving throughput and cost efficiency by up to 2.3× and 1.6×, respectively, compared to existing disaggregation methods, while generalizing to model architectures where prior approaches do not apply. Surprisingly, a heterogeneous GPU pair under FluidGPU can even exceed the throughput of two homogeneous high-end GPUs at a lower cost.


Presentation: file

Back to Technical Papers Archive Listing