Authors: Yingtao Li, Zihan Yang, and Mo Sun (Zhejiang University); Jie Zhang (Ningbo Global Innovation Center, Zhejiang University; Zhejiang University); and Kaiqi Chen, Xuzheng Chen, and Zeke Wang (Zhejiang University)
Abstract: Floating-point (FP) gradient aggregation can bottleneck distributed machine learning training. In-network aggregation (INA) can reduce this overhead by aggregating gradients in network switches. However, current in-path programmable switches based on the protocol-independent switch architecture (PISA) lack native FP support. Existing work emulates FP INA with integer arithmetic units, consuming substantial pipeline resources and limiting aggregation parallelism. We present DistFP, a host-switch co-designed system for high-parallelism FP AllReduce on PISA-based in-path programmable switches. Its key idea is a state-dependency decomposition: DistFP moves worker-local functions to the host while retaining cross-worker collective functions in the switch, allowing more concurrent switch aggregation instances under a fixed resource budget. On a real Tofino 1-based prototype, DistFP achieves 1.3x and 6.7x higher FP32 aggregation throughput than NCCL Ring and FPISA, respectively. In our end-to-end training evaluation, DistFP improves performance by up to 1.14x over NCCL Ring.