Technical Papers Archive

Low-Overhead Silent Error Detection for Large Language Model Inference at Scale


Authors: Abdullah Naveed, Mahina Sheikh, and Guanpeng Li (University of Florida)

Abstract: Large language models (LLMs) are widely deployed across large-scale computing systems, where soft errors can silently corrupt computations and degrade output quality. Existing protection techniques often rely on broad monitoring or expensive checking mechanisms, incurring non-negligible runtime overhead and limiting scalability across architectures and workloads. In this work, we propose TRACE, a lightweight fault-detection technique for generative LLMs. TRACE identifies a small set of critical layers that are most informative under fault conditions, places lightweight activation-based symptom monitors on these layers during inference, and uses a trained model to determine whether a fault has corrupted the generated output. Across 15 LLMs and three datasets, TRACE achieves 95.3% average fault-detection coverage with only 0.56% runtime overhead, a 6.1× reduction relative to prior LLM fault-tolerance techniques. TRACE further remains effective under multi-bit faults and multi-dataset profiling, demonstrating robust detection across diverse fault conditions and workloads.


Presentation: file

Back to Technical Papers Archive Listing