Workshops Archive

OPSERVE: Opportunistic LLM Inference over Fragmented GPU Capacity in HPC Systems


Workshop: AI on HPC: Performance Engineering, Challenges and Opportunities

Authors: Patrick Watters (University of Nevada, Reno); Xiaolong Ma (Argonne National Laboratory); Harry Yu (Cornell University); Ian Foster and Michael Papka (Argonne National Laboratory); Lei Yang (University of Nevada, Reno); Feng Yan (University of Houston); and Rajkumar Kettimuthu (Argonne National Laboratory)

Abstract: Large language model (LLM) inference is an important GPU workload, while batch-scheduled HPC systems often leave nodes transiently idle because available resources cannot satisfy queued jobs. Our analysis of a year-long HPC trace finds that this unused capacity accounts for 9.81% of total node-hours and changes on timescales of seconds. Exploiting such transient capacity therefore requires serving decisions to adapt to resource availability and request demand. We present OpServe, a malleable LLM serving system that combines persistent and transient inference workers, adapts worker roles between prefill and decode, and reuses completed prefill state through shared KV storage. We evaluate three request traces with Llama-3.1-8B, Qwen3-8B, and Qwen3-14B. When both OpServe and a fixed prefill/decode baseline sustain the request arrival rate, OpServe reduces p95 end-to-end latency by 1.5–2.1×. Under overload, adaptive allocation reduces queue growth and yields up to 14.3× lower observed p95 end-to-end latency.


Back to AI on HPC: Performance Engineering, Challenges and Opportunities Archive Listing

Back to Full Workshop Archive Listing