Authors: Yingtao Li, Hongjing Huang, Changyue Liao, Zihan Yang, Mo Sun, Xueying Zhu, and Jie Zhang (Zhejiang University); Bo Zhao (Aalto University); and Zeke Wang (Zhejiang University)
Abstract: Sparse AllReduce is a promising primitive for accelerating distributed machine learning training with sparse gradients, but existing designs fail to deliver high algorithm bandwidth. The root cause is their reliance on CPU-initiated coordination, which imposes high packet-processing overhead during aggregation and creates severe load imbalance across CPU cores, limiting effective exploitation of sparsity. We observe that sustaining high-throughput sparse AllReduce requires removing the CPU from both the coordination path and the fine-grained sparsification path. Guided by this insight, we present SmartReduce, a SmartSwitch-aware, SmartNIC-centric sparse AllReduce. SmartReduce combines two key mechanisms: (1) a switch-aware, NIC-initiated aggregation mechanism that performs in-network coordination between SmartNICs and the SmartSwitch with stable sub-RTT overhead, and (2) a NIC-centric fine-grained sparsification mechanism that processes fine-grained blocks in high parallelism to exploit intra-block sparsity. Experimental results show that SmartReduce at 99% sparsity achieves up to 6.4x higher algorithm bandwidth than the state-of-the-art aggregator-server-based sparse AllReduce method.