Birds of a Feather Archive

The Future of Dynamic Resource Utilization in HPC: Chances and Challenges


Authors: Martin Schreiber (Université Grenoble Alpes), Martin Schulz (Technical University of Munich), Antonio Peña (Barcelona Supercomputing Center (BSC)), Daniel Milroy (Lawrence Livermore National Laboratory (LLNL)), Dominik Huber (Technical University of Munich), Jesus Carretero (Universidad Carlos III de Madrid (UC3M)), Estela Suarez (Forschungszentrum Jülich, University of Bonn), Hans-Christian Hoppe (Forschungszentrum Jülich)

Abstract: The increasing computational demands are driving the deployment of ever-larger HPC and AI infrastructures. While these systems deliver unprecedented performance, they also pose critical challenges to their efficient use, particularly in terms of energy consumption.

This BoF will explore dynamic resource management as a key enabler of higher utilization and reduced energy consumption. Instead of traditional batch scheduling, dynamic resource control at the start or during a job’s runtime enables better allocation of compute capacity, memory, network availability, and storage. This BoF will address the two major key challenges of novel scheduling and its integration across the software stack.


Long Description: This BoF serves two main goals:

- Firstly, provide information on the limitations of the state-of-the-art in how current HPC centers utilize their systems, followed by near- and far-future perspectives on this with respect to dynamic resource utilization.

- Secondly, foster partnerships among academia, industry, and government to develop scalable approaches and open-source tools, and to ensure that the required support is integrated into key software components and standards such as MPI, OpenMP, resource management systems/software, and AI frameworks.

To make dynamic resource management state-of-the-art, we need to address a variety of complementary topics in this BoF. We ask reviewers to understand that focusing on a single topic won’t allow us to tackle this challenge. Consequently, we have to cover a variety of topics where our speakers represent worldwide leading experts:

Topic I) Infrastructures to conduct research on dynamic resource utilization: We will have three speakers, with each one of them providing insight into the developments they are involved in. D. Huber will discuss the “Dynamic Processes with PSets” (DPP) approach, A. Peña will present the “Dynamic Management of Resources” (DIMMER), and J. Perez will provide insight into FlexMPI.

Topic II) New approaches for scheduling: The flux scheduler is a next-generation workload manager also used operationally on “El Capitan”. Since dynamic resource utilization adds even greater complexity to the scheduler, we see novel approaches, such as Flux's hierarchical resource management and graph-based scheduling, as necessary. D. Milroy will give an overview.

Topic III) Monitoring systems: Dynamic resources impose new requirements for performance analysis and the scalability of monitoring infrastructures in large-scale systems. M. Schulz will address this drawing on his long-standing experience in the field, providing information on the need for monitoring to support efficient resource management in increasingly heterogeneous and dynamic computing environments.

Topic IV) Long-term operational requirements: The integration of all aforementioned components is currently addressed by the SEANERGYS EU project, presented by Hans-Christian Hoppe in this BoF. He will provide the necessary insight and understanding into the operational challenges of addressing this across an operational software stack.

Since this BoF will focus on jobs to adapt to resources at the beginning and during runtime, using moldability, elasticity, and malleability techniques, this jointly enables new ways to further optimize system utilization and energy efficiency, enabling more sustainable and effective operation of next-generation computing infrastructures. Given the current challenges of extreme energy consumption and low utilization of HPC systems, we see this BoF as highly relevant to everyone involved in HPC.

This BoF was held regularly at ISC 2024, 2025, and 2026 and was a great success, with about 100 attendees each. We adopted www.wooclap.com for our audience interaction and individual Q&A minutes for each speaker, and we plan to use it again for this BoF due to its success.

Website: https://dynreshpc.github.io/



Back to Birds of a Feather Archive Listing