Workshops Archive

Energy-Efficient Streaming FPGA Accelerator for DeepSeek Sparse Attention Decode


Workshop: Twelfth International Workshop on Heterogeneous High-Performance Reconfigurable Computing (H2RC 2026)

Authors: Jiakang Fan, Rakshith Jayanth, and Viktor Prasanna (University of Southern California)

Abstract: DeepSeek Sparse Attention (DSA) reduces long-context inference cost by scoring all cached tokens and attending only to the top-$k$. At batch one, however, GPUs execute this operation inefficiently because its dependent scan, exact selection, and irregular gather provide insufficient parallelism. For exact per-query DSA, we show that the bottleneck is primarily architectural: after eliminating host-side kernel-launch overhead, the kernel reaches at most 1.19% of GPU INT8 peak across 4K--128K contexts. We present a streaming FPGA accelerator that fuses index generation and sparse attention while retaining intermediates on chip. It reduces per-head scores immediately, finds the exact top-$k$ threshold by counting rather than ranking, and fetches each selected key/value vector once for all heads. On AMD Alveo U55C, post-route power is \SI{11.324}{\watt}; against an RTX 6000 Ada running the quantized kernel, it achieves $8.0$--$10.2\times$ lower energy and $3.9$--$6.4\times$ lower energy--delay product.


Back to Twelfth International Workshop on Heterogeneous High-Performance Reconfigurable Computing (H2RC 2026) Archive Listing

Back to Full Workshop Archive Listing