Authors: Haoxuan Wang and Chen Wang (Nanyang Technological University, Singapore)
Abstract: The deployment of long-context LLMs poses significant challenges due to the intense computational cost of self-attention and substantial memory overhead of the KV cache. We introduce HieraSparse, a hierarchical KV cache compression framework with acceleration kernels that leverage GPU Sparse Tensor Cores to speed up semi-structured KV cache attention for both prefill and decode phases. With hierarchical design, our method allows for flexible quality-sparsity trade-off and successfully converts sparsity into efficiency. Compared to state-of-the-art decode method that utilizes unstructured sparsity, HieraSparse achieves 1.2× KV compression ratio and 4.57× attention speedup at the same sparsity level. Furthermore, we extended semi-structured KV cache pruning to the prefill stage, which demonstrated up to 1.85× attention speedup at the highest sparsity. Lastly, we evaluate generation quality of HieraSparse with a simple magnitude-based pruning method, and the results show that 1.37× prefill speedup and 1.77× decode speedup can be achieved without significant quality drop.
Presentation: file