Poster Type: ACM SRC Posters
Author: Saverio Pasqualoni (King Abdullah University of Science and Technology (KAUST), Sapienza University of Rome), Marco Canini (King Abdullah University of Science and Technology (KAUST)), Daniele De Sensi (Sapienza University of Rome)
Abstract: Collective operations are a key performance factor on the Cerebras WSE-3, a wafer-scale processor with 900,000 cores on a 2D mesh. At this scale, communication distance strongly affects latency, making locality a first-order concern in the design of collective algorithms.
We develop tree-style schedules that minimize communication distances for 1D Reduce, the building block of 2D collectives and AllReduce. We port Bine trees, the locality-optimized binomial trees we recently introduced, to the WSE-3; implement balanced binary-tree reductions; and introduce unbalanced binary trees as well as two-phase variants with tree-based outer phases. Each schedule is paired with an analytical cost model and implementation-level optimizations such as fused receive-send transports. On the official cycle-accurate simulator, our schedules outperform fixed-pattern state-of-the-art algorithms by up to 1.63x.
Best Poster Finalist (BP): no
Poster: PDF
Poster Summary: PDF