Authors: Yanbo Chen, Mingqi Wang, Shuhao Zhang, Xiaofei Liao, and Hai Jin (Huazhong University of Science and Technology (HUST))
Abstract: When key-value (KV) cache demand exceeds GPU memory in online large language model (LLM) serving, engines reclaim active requests' cache. Poor victim selection increases recomputation and time to first token (TTFT). We present BidKV, a utility-guided policy that ranks victims by recoverable capacity and estimated disruption cost. BidKV integrates with vLLM and SGLang through a portable adapter that reuses native preemption. With Llama-3.1-8B-Instruct and ShareGPT traces, BidKV reduces 95th-percentile TTFT by over 90% and improves 300 ms service-level objective (SLO) attainment by 14.8 percentage points over vLLM's native policy. On SGLang, it reduces 95th-percentile TTFT by 94.6% and improves SLO attainment by 38.8 percentage points over native least-recently-used eviction.
Source code is available at https://github.com/CGCL-codes/BidKV.
Presentation: file