Workshops Archive

On preprocessing heterogeneous HPC telemetry for use in foundation models and digital twins


Workshop: Workshop on Operational Data Analytics in HPC (HPC-ODA)

Authors: Wesley Brewer (Oak Ridge National Laboratory (ORNL)), Ketan Maheshwari (Oak Ridge National Laboratory), Qing Liu (Mississippi State University), Sean Wilkinson (Oak Ridge National Laboratory), Bertrand Cirou and Gérard Gil (CINES), Jean Luca Bez (Lawrence Berkeley National Laboratory), and Kaveen Hiniduma and Suren Byna (Ohio State University)

Abstract: HPC facilities publish telemetry supporting foundation-model training and digital-twin replay but lack common fields, units, and privacy conventions. We extend REDI and AIDRIN to HPC telemetry via a new domain processor and agentic telemetry skill, and apply SetGo, forming an end-to-end pipeline that transforms heterogeneous telemetry into AI-ready and digital-twin-ready datasets. For digital twins, we demonstrate cross-facility replay fidelity: naively replaying Marconi100 telemetry through a Frontier twin violates its physical power envelope on 10.23% of samples; renormalization reduces this to 0.0042%. The pipeline supports representative replay and targeted emulation. For foundation models, we find static scheduler metadata alone gives no signal for predicting job-level details; however, ExaFM predicts job runtime with R2 = 0.960 when conditioned on a pre-anonymization identity embedding; without it, transferring that head to Marconi100, unseen during pretraining, reaches R2 = 0.716. We close with actionable recommendations for publishing AI- and digital-twin-ready telemetry toward future pretraining corpora.


Back to Workshop on Operational Data Analytics in HPC (HPC-ODA) Archive Listing

Back to Full Workshop Archive Listing