Workshops Archive

Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling


Workshop: EESP 2026: 3rd International Workshop on Energy Efficiency with Sustainable Performance: Techniques, Tools, and Best Practices

Authors: Mobina Kashaniyan and Ali Jannesari (Iowa State University)

Abstract: Test-time scaling can improve large language model reasoning by generating and combining multiple candidate responses. The inference budget is described by the number of generated candidates, N, but N tells us how many candidates are generated, not how they are executed. The same budget can come from one batched call or several sequential calls. Using Phi3-mini and Qwen2.5-1.5B on 500 GSM8K prompts, increasing N from 1 to 8 improves accuracy by 8.4 and 18.4 points, but accuracy alone does not show its system cost. Fixing N = 8, we compare four generation schedules, 1×8, 2×4, 4×2, and 8×1, measuring latency, throughput, GPU-hours, and gross GPU-device energy. On A100 GPUs, eight serial calls use 4.64–4.86× as much energy and have 5.77–6.12× the P95 latency of one batched call, confirmed across A100 nodes and SciQ/V100 experiments. Candidate count alone cannot describe system cost; evaluations should also report generation schedule and GPU-level metrics.


Back to EESP 2026: 3rd International Workshop on Energy Efficiency with Sustainable Performance: Techniques, Tools, and Best Practices Archive Listing

Back to Full Workshop Archive Listing