Workshops Archive

MiniLlama-Infra: Micro-Architecture Ablation of LLM Inference on Commodity GPU Accelerators


Workshop: AI on HPC: Performance Engineering, Challenges and Opportunities

Authors: Zijian Zhang (East China Normal University)

Abstract: Quantizing LLM weights to INT8 can reduce storage yet still slow inference when GPU access patterns are poorly matched to the hardware. We present MiniLlama-Infra, a transparent from-scratch CUDA inference engine for Llama-3.2-1B and a six-stage micro-architecture study. Archived Nsight Compute measurements on an RTX 4060 Laptop show that naive INT8 GEMV reduces memory throughput from 58.1 to 19.2 GB/s and increases latency from 289.1 to 298.5 µs. A warp-cooperative mapping improves coalescing and available parallelism together, reducing kernel latency to 52.3 µs, a 5.7× speedup. Matched Stage 5/6 experiments further show that W8A8 improves decoding throughput by 15.8% on RTX 4060 Laptop and 12.8% on A100 SXM4, with estimated GPU energy-efficiency gains of 17.0% and 8.4%. Additional A100 kernel experiments, selected historical cross-generation and 3B results, and a compiled-engine perplexity pilot extend the evaluation while explicitly bounding the scope of the claims.


Back to AI on HPC: Performance Engineering, Challenges and Opportunities Archive Listing

Back to Full Workshop Archive Listing