Technical Papers Archive

Efficient Serving for Agentic LLM Workflows via Micro-Task-Level Parallelism


Authors: Siqi Wang, Xuezhu Wang, and Hongliang Cao (Beihang University); Ruihao Gong (Beihang University, SenseTime); Qingxiao Sun (Beihang University); Yufan Xu (Independent Researcher); Huaitao Zhang (Beihang University); Wei Li (University of Sydney); Hailong Yang (Beihang University); Albert Zomaya (University of Sydney); and Depei Qian (Beihang University)

Abstract: Agentic large language model (LLM) workflows leverage the advanced capabilities of LLMs to instantiate a multi-agent paradigm, where each LLM invocation constitutes an agentic task. Existing optimizations rely on coarse-grained task-level scheduling, which fails to exploit fine-grained parallelism within the workflow, leading to resource underutilization and bursty latency. In this paper, we present Grape, a full-stack LLM system for serving agentic LLM workflows that fully exploits micro-task (μTask)-level parallelism. On the frontend, Grape proposes the TaskFlow abstraction to provide a declarative interface for workflow orchestration. Subsequently, Grape decomposes tasks represented in TaskFlow into fine-grained μTasks that expose the parallelism within the workflow. For the LLM engine, Grape adopts a μTasks-aware scheduling policy with SLO-derived constraints to ensure low-latency execution with improved resource utilization. Moreover, Grape also proposes a dedicated memory management mechanism to alleviate KV Cache memory pressure. The experimental results demonstrate the superiority of Grape over state-of-the-art solutions.


Presentation: file

Back to Technical Papers Archive Listing