Workshop: 8th International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)
Authors: Aditya Bhosale and Laxmikant Kale (University of Illinois at Urbana-Champaign)
Abstract: Exploiting discounted spot instances for HPC requires changing an application's resource allocation at runtime, shrinking ahead of an interruption and expanding onto replacement capacity. Existing mechanisms implement rescaling as a full teardown and cold restart, where restart accounts for up to 95\% of the overhead, growing with node count and, on GPUs, with CUDA context initialization. We present a no-restart rescaling mechanism for Charm++ in which surviving processes never exit: it negotiates membership with a lightweight external coordinator, reconciles UCX endpoints with the new cluster view, and returns to the top of runtime initialization while preserving live application state. A rescaling operation costs 8--15\,ms on CPUs and 7--11\,ms on GPUs at 4--32 instances, two orders of magnitude below restart-based implementations. GPU device state persists in place, eliminating the checkpointing daemons previously required. End to end, eight simultaneous interruptions cost 0.2\% of runtime on CPUs and 0.6\% on GPUs.
Back to 8th International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC) Archive Listing
Back to Full Workshop Archive Listing