Research and ACM SRC Posters Archive

Evaluating All-Reduce Operations in DDP Training with FABRIC: Framework for Architecture, Backend, and Reduction Interchange Comparison


Poster Type: Research Posters

Author: Matilde Restelli (University of Illinois Chicago), Ke Fan (Temple University), Aadirupa Saha (University of Illinois Chicago), John Holmen (Oak Ridge National Laboratory (ORNL)), Sidharth Kumar (University of Illinois Chicago)

Abstract: Distributed deep learning increasingly relies on large GPU systems, where distributed data parallel (DDP) training replicates models across devices and synchronizes gradients each step. As model sizes and GPU counts grow, gradient synchronization becomes a major overhead. Although many optimized all-reduce algorithms exist, they are often embedded in different communication libraries or stack layers, making controlled comparison in full training workflows difficult. We present FABRIC (Framework for Architecture, Backend, and Reduction Interchange Comparison), an open-source, portable, and extensible framework for evaluating PyTorch DDP across communication backends and collective strategies. Using DDP communication hooks, FABRIC integrates alternative all-reduce algorithms without modifying PyTorch internals or training code, enabling reproducible comparisons across models, datasets, and backends including MPI, NCCL, and RCCL, and letting researchers add and benchmark custom communication methods. Experiments on Frontier and Polaris reveal meaningful performance differences among synchronization strategies. The poster presents FABRIC's architecture, extensibility, and productivity, and current results.

Best Poster Finalist (BP): no
Poster: PDF
Poster Summary: PDF


Back to Poster Archive Listing