Authors: Zhongcheng Du, Yu Huang, Qihang Qiu, Yi Zhang, Deting Chen, Chencheng Ye, Xiaofei Liao, and Hai Jin (Huazhong University of Science and Technology (HUST))
Abstract: Long sequence generation is a critical workload for large language model (LLM) inference, but its high computational cost limits scalable deployment. Existing speculative decoding methods accelerate inference by verifying multiple draft tokens per step, yet uniform speculation concentrates failures in late decoding stages, where marginal payoff is highest, reducing overall speedup.
We propose DiffSpec, a differential speculative decoding framework that adapts speculation to position-specific utility. At the algorithmic level, DiffSpec employs utility-guided draft construction with relevance-aware key-value (KV) selection and adaptive draft tree construction, improving candidate quality in high-impact regions. At the system level, a differential speculative execution engine supports utility-adaptive speculation via reuse-aware KV residency and topology-aware fused verification, handling irregular KV cache accesses and heterogeneous workloads. Experiments on four NVIDIA H100 GPUs with sequences up to 128K tokens show end-to-end speedups of 7.72×, 3.45×, and 2.36× over EAGLE-3, TriForce, and MagicDec, respectively.
Presentation: file