Technical Papers Archive

Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-Scale MoE Serving


Authors: Ziming Liu (National University of Singapore); Boyu Tian, Guoteng Wang, Zhen Jiang, Peng Sun, Zhenhua Han, and Tian Tang (Shanghai Qiji Zhifeng Co., Ltd.); Xiaohe Hu, Yanmin Jia, Yan Zhang, He Liu, and Mingjun Zhang (infrawaves); Yiqi Zhang (National University of Singapore); Qiaoling Chen (Nanyang Technological University, Singapore); Shenggan Cheng (National University of Singapore); Mingyu Gao (Tsinghua University); Yang You (National University of Singapore); and Siyuan Feng (Shanghai Innovation Institute)

Abstract: Mixture-of-experts (MoE) models challenge serving infrastructures with dynamic, sparse expert utilization, causing instability on conventional systems designed for dense architectures. We propose EaaS, a novel serving system to enable efficient, scalable, and robust MoE deployment. Our system disaggregates MoE modules into independent, stateless services. This design enables fine-grained resource scaling and provides inherent fault tolerance by decoupling compute units. The architecture is powered by a high-performance, CPU-free peer-to-peer communication library that ensures minimal overhead and high throughput. Experiments confirm EaaS's scalability and efficiency, achieving performance comparable to monolithic systems while providing robust fault tolerance and strong scalability. EaaS incurs less than a 2% throughput reduction under simulated hardware failures that would otherwise halt monolithic architectures. It further saves up to 37.5% of computing resources through dynamic fine-grained adaptation to serving traffic, demonstrating strong resilience for large-scale MoE deployment in production.


Presentation: file

Back to Technical Papers Archive Listing