Authors: Peihao Yang, Teng Ma, Tinghe Zhu, Yang Su, Ziqu Yu, Zongyong Wu, Jiahao Zeng, Shuo Tan, Zheng Liu, and Tao Ma (Alibaba Cloud)
Abstract: In public cloud environments, secure containers face challenges in device virtualization due to their micro virtual machine architecture. We introduce Mercury, a user-space API remoting framework co-designed with our secure container RunD. Mercury leverages a dynamic shared memory interface between guest and host, along with KVM hypercalls for efficient notifications.
When applied to CUDA, Mercury preserves API semantics across virtualization boundaries, enabling GPU virtualization with under 4% overhead during large language model inference. Mercury also achieves a 20% speedup compared to existing API remoting solutions. Building on Mercury, we deploy GPU sharing and checkpoint/restore in Alibaba Cloud, effectively doubling GPU deployment density and reducing cold start latency by over 95% in practice.
Presentation: file