Authors: Chengyu Sun and Yaqi Xia (School of Computer Science, Wuhan University); Ruirui Pan (Central China Normal University); Donglin Yang (NVIDIA Corporation); Xiaobo Zhou (IOTSC & Department of CIS, University of Macau); and Dazhao Cheng (School of Computer Science, Wuhan University)
Abstract: Long-context LLM inference is constrained by the memory-bandwidth and capacity demands of the key-value (KV) cache. Existing sparse methods typically use fixed per-head token budgets, overlooking variations in runtime demand. Moreover, since they retain all tokens in memory, these methods do not alleviate GPU memory capacity constraints. To overcome these limitations, we present DySpin, a plug-and-play library for dynamic sparse long-context inference. Its training-free runtime allocator dynamically adjusts token budgets to preserve important tokens and improve accuracy. To address the resulting workload imbalance, FastEstimate fuses token-importance evaluation and selection to generate scheduling indices that guide a specialized decoding attention kernel for fine-grained load balancing. DySpin further employs six-bit KV cache quantization and a novel packing format that enables register-level parallel unpacking and dequantization, reducing both dequantization overhead and memory footprint. DySpin integrates with existing sparse frameworks, improving accuracy while reducing long-context inference latency and KV cache memory usage.
Presentation: file