Workshop: Twelfth International Workshop on Heterogeneous High-Performance Reconfigurable Computing (H2RC 2026)
Authors: Curt John Bansil, Kaustubh Manohar Mhatre, Aviral Shrivastava, and Aman Arora (Arizona State University)
Abstract: Consumer neural processing units (NPUs) make local large language model inference attractive, but limited resources make mapping transformer layers to them challenging. This paper categorizes and evaluates three ways to offload transformer layers on Ryzen AI NPUs. Single-operator offload (SOO) configures the NPU to execute one operation within a transformer layer. Multi-operator offload (MOO) submits an ordered list of operators to be executed sequentially on the NPU. We introduce fused-operator offload (FOO), which groups graph operators into subgraphs, maps their dataflows through the NPU, and adjusts the loop order to preserve intermediate data on-chip. Across encoder and decoder layers, FOO improves effective GFLOP/s over MOO and SOO by up to 2.22× and 6.30×, respectively. We further introduce a FOO method to speed up attention and feed-forward computations by 7.80× and 6.83×, respectively. Compared to the integrated GPU (iGPU), FOO improves energy efficiency by up to 9.15× higher effective GFLOP/s/W.
Back to Twelfth International Workshop on Heterogeneous High-Performance Reconfigurable Computing (H2RC 2026) Archive Listing
Back to Full Workshop Archive Listing