Workshops Archive

A JIT framework for Implicit GEMM and FlashAttention Micro-Kernels in RISC-V


Workshop: International Workshop on RISC-V for HPC (RISCVHPC)

Authors: Yuhao Wang (MetaX Integrated Circuits (Shanghai) Co., Ltd; Chalmers University of Technology) and Tiandu Li, Sonia Rani Gupta, and Miquel Pericas (Chalmers University of Technology, University of Gothenburg)

Abstract: The RISC-V Vector extension (RVV) is a credible CPU deep-learning inference target, but its software stack lags the silicon: optimized kernels are scarce, and prior co-design studies of RVV convolution use compile-time intrinsic kernels built on the memory-hungry im2col+GEMM formulation, evaluated almost entirely in simulation. We present a lightweight just-in-time (JIT) framework that generates register-blocked, vector-length-agnostic Implicit GEMM micro-kernels for RVV at runtime, once the true vector length and layer shape are known, and integrates them into Intel oneDNN, eliminating the im2col buffer. We characterize six vector widths (VLEN 128-8192 bits) in cycle-accurate gem5 and validate on a BananaPi-F3 board, then extend the infrastructure to JIT FlashAttention. Implicit GEMM reaches 3.28x over scalar in gem5 and 3.08x on silicon; for FlashAttention, runtime generation is fastest up to VLEN=512, the range of every shipping RVV core. An autotuner separates portable tuning knobs (tile height MR) from ISA-specific ones (LMUL, NR).


Back to International Workshop on RISC-V for HPC (RISCVHPC) Archive Listing

Back to Full Workshop Archive Listing