Technical Papers Archive

When KV Meets Embeddings: Dynamic GPU Memory Allocation for Accelerating Generative Recommender Serving


Authors: Wenjun Yu and Amelie Chi Zhou (Hong Kong Baptist University) and Shuguang Han (Alibaba)

Abstract: Generative recommender (GR) inference faces a dual memory bottleneck absent in LLM serving. Embedding hot caches and KV caches compete for limited GPU HBM, while existing systems optimize them in isolation, leading to suboptimal tail latency. We observe that (1) the optimal EMB–KV memory allocation ratio shifts by up to 0.35 across workloads, and (2) static allocations leave 20%–30% latency improvement untapped, while continuous reallocation risks cache thrashing and P99 SLO violations.

We present RACER, which jointly and adaptively allocates HBM at runtime. It addresses two challenges: (1) adaptive memory allocation via a three-layer PPO controller with online adaptation and burst recovery, achieving 32 μs decision latency; and (2) embedding-KV-aware scheduling that maximizes joint EMB–KV cache reuse across nodes.

Evaluations on three production-scale datasets show that RACER reduces P99 latency by 24%–38% over the best static strategy and achieves 93.5%–99.6% SLO satisfaction, outperforming state-of-the-art baselines (67%–94.2%) without sacrificing throughput.



Presentation: file

Back to Technical Papers Archive Listing