Technical Papers Archive

Pushing the Limits of Structured Sparse GEMM on Hopper GPUs via Analytical Modeling


Authors: Heng Shi (Shanghai Enflame Technology Co., Ltd.; Shanghai Jiao Tong University); Qiqi Gu and Chenpeng Wu (Shanghai Jiao Tong University); Xiaofeng Guan (Shanghai Jiao Tong University; Shanghai Enflame Technology Co., Ltd.); Sijie Wu, Enmin Fan, and Guoqing Bao (Shanghai Enflame Technology Co., Ltd.); and Jianguo Yao (Shanghai Jiao Tong University)

Abstract: Structured sparsity improves performance in LLM inference and scientific computing, yet Hopper 2:4 sparse GEMM remains poorly understood. Published kernel-level optimizations target older architectures, while no prior work investigates how Hopper's Tensor Memory Accelerator (TMA), warpgroup execution, expanded shared memory, and asynchronous pipelines reshape the design space. We present an analytical model that captures roofline bounds, L2 reuse, pipeline overlap, and wave quantization for Hopper sparse GEMM. It serves as a guardrail for configuration search. Guided by the model, we develop an auto-tuned kernel combining warp-specialized TMA pipelining, metadata prepacking, barrier-decomposed overlap, and per-operand swizzles. On an H800 PCIe GPU, it outperforms cuSPARSELt on 80 of 95 shapes (84%) under identical problem configurations, with mean speedup of +16.7% and up to 1.4x. The model identifies memory bottlenecks and narrows the configuration space before compilation, reducing tuning cost while retaining high-performance configurations. This paper bridges both gaps.


Presentation: file

Back to Technical Papers Archive Listing