Technical Papers Archive

ZOCheck: CPU-Shadow Checkpointing for Zeroth-Order LLM Fine-Tuning


Authors: Minqiu Sun (University of Delaware), Xin Huang (RIKEN Center for Computational Science (R-CCS)), Luanzheng Guo and Nathan R. Tallent (Pacific Northwest National Laboratory (PNNL)), Kento Sato (RIKEN Center for Computational Science (R-CCS)), and Dong Dai (University of Delaware)

Abstract: Zeroth-order (ZO) optimization is an attractive option for memory-efficient LLM fine-tuning, but its fault tolerance remains underexplored. Unlike first-order training, ZO progress can be represented by lightweight seed-and-scalar step logs, yet naive log-only recovery still incurs replay cost that grows with training progress, and shortcut replay does not preserve the executed floating-point trajectory. We present ZOCheck, a fault-tolerant ZO training system that exploits this replayable structure through a CPU shadow process that continuously replays logged updates, materializes consistent recovery images off the GPU critical path, and persists them asynchronously. ZOCheck combines non-blocking checkpointing with fast recovery from a near-current state. We also develop a cost model for choosing the snapshot policy under realistic failure rates. Experiments show that ZOCheck reduces checkpoint overhead by up to 219.7× and recovery latency by 1.55× on average over asynchronous full-state checkpointing, yielding up to 21.3× lower end-to-end wasted time while preserving exact recovery behavior.


Presentation: file

Back to Technical Papers Archive Listing