Workshops Archive

PerfDecode: Accelerating LLM Decoding on Systolic Arrays via Performance Modeling


Workshop: The 1st International Workshop on Edge-Cloud-HPC Operational Continuum (ECHO)

Authors: Dinghong Song (UC Merced, Amazon) and Rishabh Rajesh, Karthick Gopalswamy, and Patrick Lange (Amazon)

Abstract: The autoregressive nature of large language model (LLM) decoding results in matrix multiplications with small token dimensions, which underutilize systolic arrays and exhibit low arithmetic intensity. Efficient execution of these workloads requires jointly improving processing-element utilization and coordinating data movement with computation under limited on-chip memory. We present PerfDecode, a performance-model- guided approach to accelerating LLM decoding on systolic arrays. PerfDecode reorganizes matrix multiplication execution to exploit idle processing elements and pipeline data movement with computation. At its core, an analytical performance model captures DMA transfer and launch overhead, systolic-array com- putation, reduction and write-back costs, and on-chip memory constraints to automatically determine blocking configurations for different matrix shapes. On AWS Trainium2, PerfDecode consistently outperforms tiled matmul across diverse decode operators in Llama-3-8B and Llama-3-70B, achieving geometric- mean speedups of 2.73× and 2.85×, respectively.


Back to The 1st International Workshop on Edge-Cloud-HPC Operational Continuum (ECHO) Archive Listing

Back to Full Workshop Archive Listing