Authors: Yinxiao Feng (Tsinghua University); Tiancheng Chen (ETH Zürich); Yuchen Wei (Tsinghua University); Siyuan Shen (ETH Zürich); Shiju Wang, Wei Li, and Kaisheng Ma (Tsinghua University); and Torsten Hoefler (ETH Zürich)
Abstract: The exponential growth of AI models necessitates hyper-scale networking infrastructures with unprecedented cost-effectiveness and scalability. However, traditional packet-switched fat-trees are prohibitively expensive, while direct circuit-switched topologies exhibit poor scalability and limited all-to-all performance. In this paper, we present a novel reconfigurable network architecture called RailX that achieves high scalability, high performance, high flexibility, and low cost simultaneously. RailX organizes nodes and optical switches in a 2D configuration, achieving much better scalability than existing centralized circuit switching networks. We propose a novel interconnection method based on Hamiltonian decomposition theory to organize separate rail-based rings into all-to-all topologies, simultaneously optimizing both ring-collective and all-to-all communication at ultra-scale. RailX's flat switching layer interconnects over 400K high-bandwidth chips with a diameter of just 2-4 inter-node hops. It slashes costs per all-reduce and all-to-all bandwidth to under 10% and 50% of fat-trees, respectively. For MLaaS, we flexibly map diverse AI workloads and bypass hardware failures.
Presentation: file