Authors: Zhi Ma and Shigang Li (Beijing University of Posts and Telecommunications)
Abstract: Inference serving for large language models (LLMs) often relies on static resource allocation—such as fixed Top-k routing in mixture-of-experts (MoE) models—despite substantial token-level heterogeneity. This mismatch causes systematic over-provisioning, magnified in long reasoning and batched speculative decoding.
We introduce Resource Allocation via Speculative Preview (RASP), a framework that leverages predictive signals from a draft model to guide token-level allocation before target-model verification, enabling fine-grained adjustment of compute budgets to match instantaneous demand. As an instantiation, Spec-K applies RASP to MoE routing: it uses speculative preview to estimate token uncertainty and reallocates the Top-k budget dynamically via a minimalist policy, reducing expert computation, communication traffic, and activated expert count in batched verification—thereby improving throughput and even speculative acceptance.
Experiments on distributed MoE inference show that Spec-K achieves up to 1.72× end-to-end throughput improvement over a strong baseline, with negligible accuracy loss on complex reasoning and instruction-following tasks.
Presentation: file