Technical Papers Archive

FastInfer: Breaking the Low-to-Moderate Sparsity Barrier for Efficient LLM Inference


Authors: Jinliang Shi and Shigang Li (Beijing University of Posts and Telecommunications)

Abstract: As large language models (LLMs) continue to scale, unstructured pruning has become a key technique for reducing inference cost. However, to preserve model quality, pruning is typically limited to low-to-moderate sparsity (30%–60%), where existing GPU-based sparse inference frameworks often fail to deliver consistent speedups over dense baselines. This paper proposes FastInfer, a new sparse LLM inference framework in the low-to-moderate sparsity regime. FastInfer introduces a novel thread-centric encoding paradigm that rethinks sparse computation on Tensor Core Units from the perspective of per-thread register requirements. Based on this, we design the TFBE format, which enables low-overhead operand reconstruction. FastInfer integrates an affinity-aware data-movement mechanism and a triple-stage pipeline to better exploit modern GPU features and improve pipeline utilization. Compared with the state-of-the-art SpInfer, FastInfer achieves geometric-mean speedups of 1.52x (up to 2.1x) for sparse kernels and 1.27x (up to 1.42x) for end-to-end sparse inference on H100.


Presentation: file

Back to Technical Papers Archive Listing