Authors: Weiling Yang, Jianbin Fang, Dezun Dong, Chencheng Deng, Runxi He, and Junwen Zhang (NUDT)
Abstract: Irregular GEMM, where one or more dimensions are small, is common in workloads like LLM inference and vector search. On CPUs with matrix units (e.g., Intel AMX), irregular GEMMs are data-movement-bound: limited reuse and fragmented accesses let packing and cross-NUMA traffic dominate. State-of-the-art BLAS libraries are tuned for regular shapes, using coarse-grained, ahead-of-time packing and placement-agnostic parallelism assuming substantial reuse and contiguous access. We show that the gap stems from a mismatch in computation and data movement: how operands are packed and where they reside in NUMA systems. TileFlow addresses this via (1) a fine-grained packing–compute pipeline overlapping on-demand packing and memory access with computation at micro-kernel granularity, and (2) a placement-aware task assignment reducing cross-NUMA traffic under NUMA-local allocation and page interleaving without explicit placement control. On two AMX-enabled CPUs, TileFlow achieves speedups of 1.36×, 1.47×, and 1.39× for irregular GEMMs, Llama3-8B inference, and approximate nearest neighbor search.
Presentation: file