Announcing Best Paper and Best Student Paper Finalists

GPU collective communication is often optimized for bandwidth, yet many emerging workloads are latency-bound. Long-context LLM inference is a key example, where small collectives lie on the critical path of token generation, making microsecond-scale overheads impactful. In this work, we study how to approach the hardware Speed-of-Light (SoL) lower bound for GPU collectives within a scale-up network. We identify key design principles, including barrier-free synchronization and efficient use of symmetric memory and multicast. Building on NCCL’s device-side API, we develop low-latency interfaces for constructing custom kernels and implement new symmetric collectives in NCCL. Microbenchmarks show substantial latency reductions, achieving within 10% of the measured SoL bound. Integrated into applications, these kernels reduce inter-token latency and improve throughput in LLM inference, while also accelerating cuSOLVERMp, demonstrating benefits across AI and HPC workloads.

Jeff R. Hammond, NVIDIA Corporation, Finland

Torsten Hoefler, ETH Zürich, Switzerland

Anton Korzh, NVIDIA Corporation, USA

John Bachan, NVIDIA Corporation, USA

Siyuan Shen, ETH Zürich, Switzerland

Nishank Chandawala, NVIDIA Corporation, USA

Tiancheng Chen, ETH Zürich, Switzerland

Kamil Iskra, NVIDIA Corporation, USA

Sylvain Jeaugey, NVIDIA Corporation, France

Arnav Goel, NVIDIA Corporation, USA

Zhenhao He, NVIDIA Corporation, Switzerland

Stencil computation is a fundamental kernel in scientific computing, widely used in applications such as fluid dynamics and weather simulation. Although stencils are traditionally considered memory-bound and thus ill-suited for compute-oriented Tensor Cores, recent studies report substantial acceleration, creating an apparent contradiction. This paper resolves this discrepancy through a systematic performance analysis of stencil computations on Tensor Cores. We revisit stencil-to-Tensor-Core mappings, quantify the computational redundancy introduced by hardware-driven transformations, and develop an enhanced performance model that captures arithmetic-intensity shifts under temporal fusion. From this model, we derive analytical criteria that provide a first-order estimate of whether Tensor Cores will outperform, match, or underperform CUDA Cores for a given stencil and hardware setting. Our analysis identifies the performance sweet spot for Tensor Core acceleration. Evaluations on NVIDIA GPUs with state-of-the-art implementations, including EBISU, ConvStencil, and SPIDER, validate our criteria and show its value for guiding stencil optimization.

Qiqi Gu, Shanghai Jiao Tong University, China

Haibing Guan, Shanghai Jiao Tong University, China

Chenpeng Wu, Shanghai Jiao Tong University, China

Jianguo Yao, Shanghai Jiao Tong University, China

Heng Shi, Shanghai Enflame Intelligence Technologies Co. Ltd, China

Temporal graph neural networks (T-GNNs) play a key role in modeling dynamic graphs by learning representations that capture evolving structures and interactions. However, scaling T-GNNs remains challenging, as massive real-world graphs and complex temporal dependencies incur prohibitive runtime and memory costs. Existing systems often preload entire datasets into RAM and rely on RAM–GPU pipelines, suffering from batch-boundary pipeline bubbles, thereby limiting throughput. In this paper, we present DOLPHIN, a scalable Disk–RAM–GPU pipelined system for massive T-GNNs. DOLPHIN groups training batches into chunks and incorporates two key techniques: an I/O-efficient feature window scheduler for just-in-time Disk-to-RAM chunk provisioning, and a dependency-graph–based parallel execution plan that improves GPU utilization while preserving model quality. Together, these techniques improve pipeline efficiency across the Disk-RAM-GPU memory hierarchy and computation phases. Experiments on six real-world graphs show DOLPHIN delivers up to \textbf{21.07$\times$} faster training and reduces RAM usage by up to \textbf{92.04\%} compared to state-of-the-art baselines.

Wenbo Zhen, University of Science and Technology of China, China

Jianliang Xu, Hong Kong Baptist University, Hong Kong

Xike Xie, University of Science and Technology of China, China

Rui Guo, University of Science and Technology of China, China

Zezhong Ding, University of Science and Technology of China, China

Junlin Lv, University of Science and Technology of China, China

Existing fast GPU error-bounded lossy compressors achieve high throughput but limited compression ratios because they use a fixed first-order predictor on independent blocks. We propose FSZ, which redesigns the prediction stage with three mutually reinforcing innovations within a single CUDA kernel: (1) cross-block prediction state carries Lorenzo prediction across block boundaries, eliminating 7 of 8 boundary residuals; (2) per-tile adaptive multi-order prediction and centering selects the best strategy per tile from first-order, second-order, and centering variants; and (3) single-pass four-way evaluation exploits a mathematical property of finite differences to evaluate all variants from a single data read. Experiments on NVIDIA GH200 with 8 real-world HPC datasets show that FSZ outperforms cuSZp-P by up to 10.95x and the state-of-the-art cuSZp-O by up to 2.92x in compression ratio, while simultaneously achieving the highest throughput (676 GB/s compression, 785 GB/s decompression) among all evaluated compressors.

Jiajun Huang, University of South Florida, USA

Production HPC systems generate massive volumes of heterogeneous logs that encode rich signals about failure propagation across hardware and software layers. In this paper, we present a framework for causal discovery in HPC logs that combines three complementary methods: Transformer-based counterfactual intervention, PCMCI+, and transfer-entropy. Because production environments lack labeled ground truth, our framework uses cross-method triangulation to assign confidence to discovered causal relationships. We further introduce an architecture-aware AI-agent that evaluates the physical plausibility of these relationships and enriches them with mechanism-level explanations, diagnostic commands, and severity assessments grounded in Frontier system documentation. We apply the framework to 18.4 billion log events from the Frontier supercomputer and construct a structured taxonomy of 87 event types spanning diverse hardware and software subsystems. Our results reveal causal relationships across HPC-subsystems, including rare failure pathways and multi-hop cascade chains. This work provides a scalable path toward architecture-grounded root-cause diagnosis in production HPC-systems.

Awais Khan, Oak Ridge National Laboratory, USA

Christopher Zimmer, Oak Ridge National Laboratory, USA

Anjus George, Oak Ridge National Laboratory, USA

Ahmad Maroof Karimi, Oak Ridge National Laboratory, USA

Feiyi Wang, Oak Ridge National Laboratory, USA

Woong Shin, Oak Ridge National Laboratory, USA

We propose DySpin, a plug-and-play library designed to advance Dynamic Sparse long-context inference. DySpin introduces a training-free runtime allocation strategy that dynamically adjusts token budgets, preserving critical tokens that fixed-budget methods would otherwise discard, thereby improving model accuracy. To mitigate the attention kernel workload imbalance inherently caused by runtime allocation, we develop FastEstimate, a fused kernel that integrates token importance evaluation and selection to efficiently generate dynamic scheduling indices. These indices then guide a specialized decoding attention kernel, ensuring fine-grained, load-balanced computation. To tackle memory capacity constraints with accuracy preservation, DySpin incorporates 6-bit KV cache quantization. By enabling register-level parallel unpacking and dequantization through a novel packing format, DySpin significantly alleviates the severe dequantization overhead during attention computation. DySpin can be seamlessly integrated into existing sparse frameworks, delivering superior model accuracy and lower latency for long-context inference.

Chengyu Sun, Wuhan University, China

Dazhao Cheng, Wuhan University, China

Xiaobo Zhou, University of Macau, China

Yaqi Xia, Wuhan University, China

Donglin Yang, NVIDIA Corporation, USA

Ruirui Pan, Central China Normal University, China

Low-Rank Adaptation (LoRA) has produced over 100,000 adapters for Large Language Models, driving demand for efficient multi-adapter serving. All adapters share the same expensive base-model computation, creating a natural opportunity for batching. Existing serverful systems batch across adapters for high throughput but sacrifice per-tenant isolation and elastic scaling. Serverless systems provide both, but at the cost of redundant computation and cross-adapter batching efficiency. We present SwarmLoRA, a disaggregated serverless architecture that resolves this tradeoff by batching shared computation across isolated functions. A persistent aggregator executes a single GEMM for all active requests, while lightweight workers hold only per-adapter state. SwarmLoRA achieves 4.4-5.6x higher throughput than ServerlessLoRA and ServerlessLLM; under trace replay, it completes 98-100% of requests, where baselines complete fewer than one-third. Cold starts take 619 ms (2.9x faster than the fastest baseline) with 95 ms adapter swaps. SwarmLoRA delivers serverful batching efficiency with serverless isolation.

Mausam Basnet, University of North Texas, USA

Tong Shu, University of North Texas, USA