Technical Papers Archive

LRCCL: Rethinking Collective Communication for Long-Haul RDMA Networks


Authors: Yuke Li (University of California, Merced) and Zheng Cao and Xiaoyi Lu (University of Florida)

Abstract: Long-haul RDMA links operate at millisecond-scale round-trip times that break the assumptions baked into collective communication stacks designed for microsecond-scale-datacenter networks. We rethink the collective communication stack for this regime and present LRCCL, a long-haul-RDMA-based collective communication library co-designed across three layers.

First, through the first systematic characterization of collective communication on the real-world ESnet long-haul RDMA, we identify three dimensions that govern collective performance: inter-node round count, per-round data scaling, and inter-rank timing skew induced by mixed intra/inter-node work. Second, we propose a split-plane transport design specifically for long-haul RDMA, with a software reliability combining automatic repeat request and forward error correction. Third, we identify long-haul-friendly collective algorithms guided by the three dimensions. On a real ESnet long-haul testbed, LRCCL achieves 4.2x-22.6x speedup over MPICH+UCX's default RC transport under the same collective algorithm and 2.0x-3.4x end-to-end speedup on NPB and LLM workloads, including under injected moderate and burst loss.




Back to Technical Papers Archive Listing