Authors: Jing Gu, Zhuang Xing, Yiheng Yang, Bowen Lv, and Jiale Wang (Huazhong University of Science and Technology (HUST)); Shuo Yuan (Nanjing University of Aeronautics and Astronautics); Zijin Chen and Jin Zhao (Huazhong University of Science and Technology (HUST)); Pengfei Zuo (Individual Researcher); and Long Zheng, Xiaofei Liao, Hai Jin, and Qinbin Li (Huazhong University of Science and Technology (HUST))
Abstract: LLM agents are transitioning from single-node to large-scale, multi-tenant services. A user request no longer corresponds to a single inference call, but to a long-running workflow interleaving heterogeneous tasks. Existing agent systems struggle with these workloads due to resource-blind, coarse-grained placement or ignore agentic workload awareness.
We present Maze, a distributed agent framework bridging the abstraction gap from agentic logic to fine-grained task-level deployment. At its core, Maze anchors task resource semantics and introduces Heterogeneity-Aware Criticality Scheduler, a scheduler leveraging agentic workload features and value-density scoring to prioritize valuable agentic workflows and critical tasks while eliminating cross-resource head-of-line blocking. Coupled with elastic inference engine scaling, zero-VRAM optimizations, and closed-loop fault tolerance, Maze ensures a robust, performant runtime. Evaluation on diverse workloads demonstrates that Maze achieves up to 2.9x higher throughput, reduces P95 latency by 54.2%, and increases GPU utilization by 50% compared to state-of-the-art frameworks.
Presentation: file