Workshops Archive

Performance Characterization of GPU Sharing Strategies for LLM Inference


Workshop: AI on HPC: Performance Engineering, Challenges and Opportunities

Authors: Madoka Takato (Ochanomizu University); Akira Hirai, Reika Kinoshita, and Masahiro Miwa (Fujitsu Limited); and Masato Oguchi (Ochanomizu University)

Abstract: With the widespread adoption of large language models (LLMs), demand has grown for LLM inference services that run continuously within High-Performance Computing (HPC) environments. Because inference requests vary over time, GPU resources dedicated to a single model may remain idle during periods of low demand. Sharing these resources among multiple inference workloads can improve GPU utilization. This paper compares two approaches for concurrently serving multiple models on a single NVIDIA H100 PCIe GPU: software-level GPU sharing using vLLM and hardware-level GPU partitioning using vLLM in combination with Multi-Instance GPU (MIG). We evaluate both approaches while varying the number of concurrently running models and the offered load. The results show that MIG-based GPU partitioning provides stable performance and slightly outperforms software-based GPU sharing when all GPU instances are fully utilized. In contrast, software-based GPU sharing exploits otherwise idle GPU resources, achieving higher throughput and lower latency when fewer models run concurrently.


Back to AI on HPC: Performance Engineering, Challenges and Opportunities Archive Listing

Back to Full Workshop Archive Listing