Authors: Evangelos Georganas, Dhiraj Kalamkar, Alexander Heinecke, and Pradeep Dubey (Intel Corporation)
Abstract: The advent of ultra-low-bit LLM models, approaching the perplexity and task accuracy of their full-precision counterparts, is ushering in a new era of LLM inference. While these advances promise models that are cost-effective in latency, memory, throughput, and energy consumption, the runtime efficiency for deploying ultra-low-bit models remains under-explored. We take a bottom-up approach: we first implement 2-bit microkernels for modern CPUs, achieving close-to-roofline performance. We integrate these microkernels into LLM inference pipelines and present end-to-end results with 2-bit models, outperforming the state-of-the-art (SOTA) bitnet.cpp runtime by 2.2x, and deliver up to 7x speedup over 16-bit inference. We extend this work to Intel Xe2 GPUs where we implement mixed-precision 2-bit kernels, and show their performance to be close-to-optimal. Integrated into the vLLM framework, these kernels deliver up to 6.7x speedup over the 16-bit end-to-end pipeline across a range of models and Xe2 GPUs, thus pushing the envelope of LLM inference.