Authors: Chao Wang (The Chinese University of Hong Kong, Shenzhen); Pengfei Zuo and Zhangyu Chen (Huawei); Yunkai Liang (Sun Yat-sen University); Zhou Yu (Huawei); and Ming-Chang Yang (The Chinese University of Hong Kong, Shenzhen)
Abstract: Large language model (LLM) serving has long centered on a debate between prefill-decode (PD) aggregation and disaggregation. We systematically compare them under joint time-to-first-token (TTFT) and time-per-output-token (TPOT) service-level objectives (SLOs): aggregation wins when TTFT SLOs are tight and TPOT SLOs relaxed; disaggregation wins in the opposite regime. Under balanced SLOs, however, both are significantly suboptimal in SLO-compliant throughput (goodput), revealing a previously uncharacterized gap.
We attribute this gap to unexploited latency slack: many requests complete well within SLOs while others approach violation. We propose latency shifting, a design principle for reallocating SLO slack across requests and phases to maximize goodput. TaiChi, an LLM serving system, unifies aggregation and disaggregation through hybrid-mode inference to realize this principle through flowing decode and length-aware prefill scheduling. TaiChi improves goodput by up to 40% and reduces P90 TTFT and TPOT by up to 5.3× and 1.6×, respectively, over state-of-the-art aggregation and disaggregation systems.