Authors: Ahmad Maroof Karimi, Awais Khan, Anjus George, Woong Shin, Christopher Zimmer, and Feiyi Wang (Oak Ridge National Laboratory (ORNL))
Abstract: Production HPC systems generate massive volumes of heterogeneous logs that encode rich signals about failure propagation across hardware and software layers. In this paper, we present a framework for causal discovery in HPC logs that combines three complementary methods: transformer-based counterfactual intervention, PCMCI+, and transfer-entropy. Because production environments lack labeled ground truth, our framework uses cross-method triangulation to assign confidence to discovered causal relationships. We further introduce an architecture-aware AI-agent that evaluates the physical plausibility of these relationships and enriches them with mechanism-level explanations, diagnostic commands, and severity assessments grounded in Frontier system documentation. We apply the framework to 18.4 billion log events from the Frontier supercomputer and construct a structured taxonomy of 87 event types spanning diverse hardware and software subsystems. Our results reveal causal relationships across HPC-subsystems, including rare failure pathways and multi-hop cascade chains. This work provides a scalable path toward architecture-grounded root-cause diagnosis in production HPC-systems.