Workshops Archive

Secure integration and scaling for thousands of Jupyter sessions across hybrid cloud and HPC resources on Anvil


Workshop: 8th International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)

Authors: Erik Gough and LJ Lumas (Purdue University)

Abstract: The Anvil Notebook Service is a highly scalable Kubernetes-based JupyterHub platform on Purdue University's NSF-funded Anvil supercomputer. We present technical solutions bridging Anvil's 1000-node Slurm HPC partition and its Kubernetes cloud resource to scale JupyterLab, VSCode and Linux Desktops for thousands of users. To scale beyond the capacity of the Kubernetes infrastructure, a locally developed, open-source software called Dynamic Node Manager converts idle HPC nodes to Kubernetes nodes on demand. To enable access to shared HPC filesystems from Jupyter pods, custom user identity integration allows containers to run as the logged-in user's UID. Users can access HPC Slurm queues for batch job submission. For efficient GPU utilization, users can dynamically slice GPUs based on vRAM requirements. Users can also run Apptainer containers and access AI agents and MCP tools. This talk shares the engineering insights for the hybrid HPC/cloud container orchestration of the service.


Back to 8th International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC) Archive Listing

Back to Full Workshop Archive Listing