Authors: Tianle Zhong (University of Virginia); Mao Lin (University of California, Merced); Hao Wu and Keren Zhou (George Mason University (GMU)); and Geoffrey Fox (University of Virginia)
Abstract: Large language model (LLM) serving systems have become critical cloud workloads, yet their performance diagnosis remains challenging due to rapid software-hardware evolution, dynamic request patterns, and the limitations of existing profiling tools. These gaps force developers to rely on manual heuristics for performance optimization, leading to slow progress and error-prone code. We present LLMProf, a low-overhead profiling framework tailored for various LLM serving systems. LLMProf employs a top-down, multi-level analysis approach that captures only the most relevant metrics at each profiling level to balance detail with minimal performance impact while remaining agnostic to frameworks and hardware. We present case studies that leverage LLMProf to optimize real-world LLM serving workloads, from which we derive a performance diagnosis taxonomy that provides actionable guidance and enables automated postmortem analysis. Our case studies demonstrate that by applying simple optimizations uncovered by LLMProf, we can achieve up to 2.09× throughput improvement.
Presentation: file