Workshops Archive

Novel Profiling Pipeline for Replay-Based Performance Analysis


Workshop: 8th Workshop on Programming and Performance Visualization Tools (ProTools)

Authors: Didem Unat, Javid Baydamirli, Mohammad Issa, and Ilyas Turimbetov (Koç University, Turkey) and Magnus Strengert (NVIDIA)

Abstract: Replay-based profilers collect hardware metrics by executing the same code multiple times with different metric groups. While this overcomes hardware-counter limits, it breaks down for multi-GPU applications: interdependent kernels in NCCL and NVSHMEM require ranks to progress together but independent replay passes are susceptible to divergence. We present a coordinated pipeline for replay-based profiling of multi-GPU applications. To preserve correctness, we introduce a profiler-side synchronization mechanism and a heuristic kernel-grouping algorithm that replays dependent kernels in lockstep across ranks. We introduce a bundling-based metric distribution algorithm that partitions metric sets across profiler instances. The pipeline merges per-process reports into a unified profile and supports report clustering for scalable exploration. We implement the profiling pipeline in NVIDIA NCU and evaluate our metric distribution implementation. Replay coordination eliminates deadlocks in interdependent workloads, while metric distribution reduces the maximum replay count by up to 3.50× versus single-GPU profiling with profiling data quality preserved.


Back to 8th Workshop on Programming and Performance Visualization Tools (ProTools) Archive Listing

Back to Full Workshop Archive Listing