Research and ACM SRC Posters Archive

Plexus-v2: Extreme-Scale GNN Training with Communication-Free Sampling and 4D Hybrid Parallelism


Poster Type: Research Posters

Author: Cunyang Wei (University of Maryland, College Park), Siddharth Singh (NVIDIA Corporation), Aishwarya Sarkar (Iowa State University), Daniel Nichols (Lawrence Livermore National Laboratory (LLNL)), Aditya Ranjan (Google LLC), Sayan Ghosh (Pacific Northwest National Laboratory (PNNL)), Ali Jannesari (Iowa State University), Nathan Tallent (Pacific Northwest National Laboratory (PNNL)), Abhinav Bhatele (University of Maryland)

Abstract: GNNs are widely used for learning on graph datasets derived from various real-world scenarios. Learning from extremely large graphs requires distributed training, and mini-batch training is a popular approach for parallelizing GNN training. Existing distributed mini-batch approaches have significant performance bottlenecks due to expensive sampling methods and limited scaling when using data parallelism. We present Plexus-v2, a four-dimensional hybrid parallel framework for scalable mini-batch GNN training that combines communication-free distributed sampling, three-dimensional parallel matrix multiplication, and data parallelism. Plexus-v2 introduces a uniform vertex sampling algorithm, enabling each process to construct its local shard of subgraph without any inter-process communication. Combining 3D PMM with data parallelism enables Plexus-v2 to scale to much larger GPU counts than vanilla data parallelism with significantly lower communication overheads. We evaluate Plexus-v2 on five graph datasets and scale to 2,048 GPUs/GCDs. On Perlmutter, Plexus-v2 achieves 3.5x end-to-end training speedup over the best-performing SOTA frameworks on ogbn-products.

Best Poster Finalist (BP): no
Poster: PDF
Poster Summary: PDF


Back to Poster Archive Listing