Workshop: FTXS: Workshop for Faults, Trustworthiness, and eXplainability for AI Systems at Scale
Authors: Hailong Jiang (Youngstown State University), Tongping Liu (Teyon AI)
Abstract: For two decades, fault tolerance has meant recovery: checkpointing and restart, rollback recovery, consistent cuts — mechanisms that assume the fault announces itself. Something crashes, hangs, or times out; recovery restores a good state and continues.A growing and expensive class of failures in large-scale AI training does not announce itself. The job runs to completion, the loss curve looks healthy, and the model is nonetheless wrong: a parameter declared trainable never receives a gradient; a resume restores weights but not optimizer state; a precision policy silently degrades. These faults waste thousands of GPU-hours and surface days later, discovered by working backwards from an evaluation metric. For this class, recovery is worse than insufficient — applied blindly, it faithfully carries the defect forward across every restart.Such failures require explanation, not just recovery, and explanation requires evidence rather than inference. Telemetry-based root-cause analysis reasons from surface signals about what probably happened; deterministic record-and-replay makes the execution itself available for interrogation.This talk argues that the fault-tolerance community's next primitive is a queryable evidence base. I ground the argument in two prior systems: iReplayer (PLDI 2018), which showed that deterministic replay of production multithreaded programs is affordable and can be performed in place, and an OOPSLA 2020 system that automated root-cause diagnosis over such recordings using replay, hardware watchpoints, and binary analysis. I then examine what large-scale AI workloads change — a semantic gap between bytes and the logical entities of a model; distribution across ranks, where each rank's evidence is self-consistent yet insufficient; and silence, which removes the trigger these mechanisms assume — and describe early work on a semantic attribution layer addressing the first. Throughout, I am explicit about what such evidence establishes and what it does not: provenance is not proof of value correctness.
Back to FTXS: Workshop for Faults, Trustworthiness, and eXplainability for AI Systems at Scale Archive Listing
Back to Full Workshop Archive Listing