Workshops Archive

Evaluating Fused Weight-Decompression General Matrix Multiplication for Quantized LLM Inference in SYCL


Workshop: Thirteenth Workshop on Accelerator Programming and Directives (WACCPD 2026)

Authors: John Park (OREISE/USDA-Agricultural Research Service)

Abstract: General matrix multiplication (GEMM) constitutes major computation load in Large Language Model (LLM) inference. Although weight quantization is the de facto standard of modern LLM inference, the GEMMs are often sub-optimized for non-CUDA backends, such as SYCL. A fused weight-decompression GEMM could potentially raise throughput via removing intermediate VRAM traffic. This paper evaluates the efficacy of the fused kernel for INT8 weight-quantized LLM inference in SYCL: We conduct standalone microbenchmarks to isolate the GEMM throughput and accuracy, followed by an inference-engine integration of the fused path that demonstrates its end-to-end gain. At the kernel level, the fused path is 1.20–1.36× faster than the baseline in GEMM. In LLM inference, the fused path raises prefill throughput by 20–57%. It closes the prefill gap against a CUDA backend from over 40% to within 7–26% where the model fits both devices, and surpasses CUDA where the Intel GPU's larger memory avoids layer-offloading.


Back to Thirteenth Workshop on Accelerator Programming and Directives (WACCPD 2026) Archive Listing

Back to Full Workshop Archive Listing