Birds of a Feather Archive

Designing Open, Heterogeneous System Monitoring Around OpenCHAMI


Authors: Massimo Benini (Swiss National Supercomputing Centre (CSCS)), John Stile (Lawrence Berkeley National Laboratory (LBNL)), James Brandt (Sandia National Laboratories), Matthew Phillips (Los Alamos National Laboratory (LANL)), Mark Plagge (Sandia National Laboratories), Sarra Refai (Jülich Supercomputing Centre (JSC))

Abstract: While modern management platforms like OpenCHAMI streamline cluster orchestration, they do not inherently provide the comprehensive monitoring and data collection required to observe every facet of a next-generation data center. Operating heterogeneous, large-scale systems requires a unified, open-source observability framework capable of aggregating distributed logs, metrics, and traces across thousands of bleeding-edge compute nodes. This BoF session brings together HPC systems engineers, data center architects, and site reliability engineers to discuss the architecture, integration challenges, and best practices for building robust telemetry pipelines around OpenCHAMI, ensuring operational visibility, rapid fault isolation, and open-source compliance in next-generation supercomputing environments.

Long Description: Goal and Topic: The Birds of a Feather (BoF) session aims to establish a collaborative community dialog focused on designing and deploying an open-source observability layer for the Open Composable Heterogeneous Adaptable Management Infrastructure (OpenCHAMI). As high-performance computing (HPC) sites increasingly transition from monolithic, vendor-locked system management tools toward flexible, microservices-based orchestrators, telemetry requirements drastically shift. This session will explore how to integrate modern, cloud-native observability frameworks—incorporating distributed tracing, structured logging, and scalable metric collection—directly into OpenCHAMI-managed environments. Relevance to the HPC Audience: Managing next-generation supercomputers demands real-time insights into system health, hardware performance, and workflow orchestration. OpenCHAMI provides a modular, zero-trust framework for machine deployment, but efficiently scaling it across thousands of high-density nodes -- with diverse compute, memory, and network technologies -- and effectively interacting with emerging accelerator systems, such as quantum and neuromorphic computers, will require a highly integrated and extensible telemetry pipeline. Attendees from national laboratories, Tier-1 data centers, and academia face shared challenges: tracking containerized microservice health, capturing low-level hardware and software performance metrics, and correlating system events with workload scheduler states. By addressing OpenCHAMI telemetry, this session directly addresses the needs of operators looking to deploy open-source system management without sacrificing operational visibility. The discussion will feature foundational architectural approaches, such as leveraging open standards like OpenTelemetry, scalable time-series storage engines, and dynamic visualization platforms, to continuously observe and maintain the intricate state of heterogeneous accelerated clusters. Expected Outcomes: The primary outcome of this BoF is to catalyze a dedicated OpenCHAMI Observability Working Group. By connecting sites currently evaluating or prototyping OpenCHAMI with telemetry experts, we expect to achieve several critical milestones: 1. Identify Core Telemetry Requirements: Map out the specific logging, tracing, and metric collection needs unique to OpenCHAMI microservices and bare-metal provisioning paths. 2. Formulate Architectural Blueprints: Share early site deployment experiences and conceptual designs for telemetry pipelines that maintain open-source compliance and minimize performance overhead. 3. Foster Vendor-Agnostic Collaboration: Create an open channel between data center architects, software maintainers, and hardware vendors to ensure future OpenCHAMI releases expose standardized, consumption-ready telemetry endpoints. Ultimately, this session will help bridge the gap between cloud-native site reliability engineering (SRE) methodologies and large-scale HPC operations, creating a clearer roadmap for facilities aiming to embrace OpenCHAMI as their primary management ecosystem.

Back to Birds of a Feather Archive Listing