Workshops Archive

Green Shift: Resize the GPU, Not the Process


Workshop: 6th International Workshop on RESource DISaggregation in High Performance Computing (RESDIS)

Authors: Aditya Dhakal, Gourav Rattihalli, Joseph Wright, Harumi Kuno, and Dejan Milojicic (HPE)

Abstract: GPUs are multiplexed across workloads using NVIDIA MPS, MIG, and CUDA Green-Contexts. Multiplexing maximizes utilization while isolation keeps tail latency predictable. But MPS and MIG are inefficient: resizing a partition requires restarting the bound processes and reloading model weights and state, incurring seconds of idle downtime that grows with model size. This poorly fits modern inference, where many models share a GPU and their compute demands shift constantly.

Green-Contexts instead change a process’s Streaming-Multiprocessors (SMs) budget on the fly, with no restart or reload, lowering reconfiguration time and GPU idle gaps. However, they target intra-application resource management, lack notion of multi-tenancy, and are not natively supported in PyTorch/vLLM. We introduce Green Shift, which repurposes Green Contexts for multi-tenant inference. On an NVIDIA H100, Green Shift cuts a large model’s reconfiguration downtime from roughly 30s to ~3s (>10× reduction), and uses ~1.5 to 2× less energy during the resize than MPS.


Back to 6th International Workshop on RESource DISaggregation in High Performance Computing (RESDIS) Archive Listing

Back to Full Workshop Archive Listing