Authors: Abhimanyu Bambhaniya (Georgia Tech, InfraVana); Hanjiang Wu (Georgia Tech); Suvinay Subramanian (Google LLC); Sudarshan Srinivasan and Souvik Kundu (Intel Corporation); Amir Yazdanbakhsh (Google LLC); Midhilesh Elavazhagan and Madhu Kumar (Intel Corporation); Minlan Yu (Harvard University); and Arijit Raychowdhury and Tushar Krishna (Georgia Tech, InfraVana)
Abstract: Modern LLM serving spans multi-stage pipelines, including RAG retrieval and KV cache reuse, with distinct compute, memory, and latency demands. Inference engines expose vast configuration spaces with no systematic navigation methodology, and exhaustive benchmarking can exceed $40K in cloud costs. Meanwhile, hardware is diversifying across AMD GPUs, TPUs, and custom ASICs, yet cross-vendor prefill-decode (PD) disaggregation lacks a unified software stack for end-to-end evaluation. We present MIST, an event-driven simulation framework that models the full serving stack across AI workload, system and software, and hardware layers, with pluggable accelerator models enabling cross-vendor evaluation. MIST achieves end-to-end fidelity within 6% of real deployed systems. In design-space exploration case studies, under stated pricing and hardware assumptions, multi-vendor PD-disaggregated deployments identified by MIST yield up to 49.3% higher tokens/$ than single-vendor PD disaggregation. MIST also enables KV storage co-design, revealing how memory hierarchy choices shape tail latency.
MIST is available at https://github.com/MIST-Simulator/MIST.