Workshops Archive

Faster Post-Outage Testing at Full Machine Scale


Workshop: HPC Systems Professionals Workshop (HPCSYSPROS26)

Authors: Rory Kelly, Ben Matthews, and Jenett Tillotson (NSF NCAR)

Abstract: System testing after outages is a standard workflow across HPC centers. When returning from an unplanned outage, the main focus is identifying any degraded hardware and quickly returning the system to production. We have developed a post-outage testing tool called pray4us that checks the full machine with component-wise tests, provides a simple to interpret pass/fail answer, provides specific actions to take if failures are encountered, and runs end-to-end at full scale with all tests in a short wall-clock time. We cover the pray4us tool and discuss using it to investigate a Slingshot-11 bandwidth problem uncovered by the tool. Through the tool we have discovered the bandwidth problem is correlated with message size suggesting a stall event rather than a degraded network component. This may be related to enabling Prioirty Flow Control on our Slingshot-11 network, so next steps for our tool includes recording NIC level counters looking for pause frames.


Back to HPC Systems Professionals Workshop (HPCSYSPROS26) Archive Listing

Back to Full Workshop Archive Listing