Birds of a Feather Archive

Open Source and Sovereign Infrastructure for HPC and AI


Authors: Stig Telfer (StackHPC Ltd, OpenStack Scientific SIG), Jonathan Mills (NASA Goddard Space Flight Center), David Power (NScale), John Michael Lowe (Indiana University), Sadaf Alam (University of Bristol), Wojciech Turek (Cambridge University)

Abstract: Open source has long been at the heart of HPC infrastructure management. Traditional HPC platforms have been well served by software stacks designed to meet this specific purpose.

A new era brings new demands for HPC and AI. Open source infrastructure management faces new challenges on many fronts.

This session features leading practitioners of a new generation of software-defined open HPC-AI infrastructure. The session will present a diverse range of open source solutions for current use cases, including advantages and disadvantages. Common problems will be identified and solutions shared. Attendees will be invited to share their experiences.


Long Description: Open source has long been at the heart of HPC infrastructure management. Traditional HPC platforms have been well served by effective software stacks, many designed well to meet this specific purpose.

We are now in a new era, and facing new demands. Open source infrastructure management faces new challenges on many fronts.

- The AI-fuelled revolution in hardware design has resulted in systems of immense complexity and cost, from which maximum effective output must be extracted to achieve a viable return on investment.

- Moving from conventional HPC systems into the age of containerization and virtualization adds new complexities in the infrastructure software stack.

- Complex workflows and innovation in application software stacks result in impossible demands for a common software runtime environment.

- Multi-tenancy, self-service and trusted research environments introduce new tensions between performance, security and flexibility.

This session brings together leading practitioners of a new generation of open source HPC-AI infrastructures. The BoF leads represent a diverse range of solutions and a diverse range of workloads. The session will present a range of open source solutions for current use cases, including advantages and disadvantages. Common problems will be identified and solutions shared. Attendees will be invited to share their experiences.

Open source is the de facto solution for sovereign infrastructure, providing technical, financial and operational sovereignty. However, it poses its own set of unique benefits and challenges. This BoF is aimed at architects, administrators, software engineers, and scientists interested in designing and deploying cloud-native open infrastructure solutions to run HPC and AI workloads.


Back to Birds of a Feather Archive Listing