Technical Papers Archive

Understanding Reliability Bottlenecks in LLM Agents for HPC I/O Telemetry Analysis


Authors: Chris Egersdoerfer (University of Delaware); Orcun Yildiz, Kevin Harms, Robert Latham, and Robert Ross (Argonne National Laboratory); and Dong Dai (University of Delaware)

Abstract: High performance computing (HPC) facilities collect extensive I/O telemetry, but turning it into operator-relevant insight remains a manual, expert-driven process. Tool-using LLM agents could help automate this workflow, but their reliability on realistic telemetry-analysis tasks remains unclear. We study this using CIODI-Bench, a benchmark of 49 expert-curated tasks built from more than two years of public production telemetry from the Polaris supercomputer. Using CIODI-Bench, we compare tool-using agents for HPC I/O telemetry analysis under controlled conditions. We find that the main reliability bottleneck depends on task structure: on constrained extraction tasks, curated semantic grounding improves performance from 73.0% to 93.5%; on data-analysis tasks, process-aware review improves performance by 13.2% over baseline; but on insight-generation tasks, the added orchestration yields no improvement while increasing output cost by nearly 3x. Our analysis results clarify the reliability limits of current agentic approaches and offer practical guidance for building more trustworthy telemetry-analysis agents.



Back to Technical Papers Archive Listing