Workshops Archive

Sharing a Fabric with Collective Communication: Two Storage Penalties in DL Training


Workshop: PDSW’26: The 11th International Parallel Data Systems Workshop

Authors: Chen Wang and Wenzhao Wu (Nanyang Technological University, Singapore) and Hyojin Kim and Jae-Seung Yeom (Lawrence Livermore National Laboratory)

Abstract: Distributed DL training on HPC systems often shares one network fabric for collective communication and parallel-filesystem I/O. Using a real GNN workload on a Slingshot-11 system, we identify two costs. First, shared-fabric I/O causes heavy-tailed DataLoader stalls: typical waits are 15 ms but spike to seconds in 28% of Lustre iterations and 12% of VAST iterations. Second, traffic-class contention slows collective communication: Lustre stalls all-reduce by up to 145 times in isolation. The first effect affects any shared-fabric storage path, whereas the second occurs only when storage and collective traffic share a traffic class. A dedicated all-flash tier on a separate traffic class substantially reduces the secondary effect but leaves its DataLoader tail heavy. Nimbus, which stages data to node-local NVMe, eliminates both by removing storage I/O from the shared fabric. Nimbus achieves 7.4 times higher throughput than Lustre and 1.31 times higher throughput than VAST with a warm cache.


Back to PDSW’26: The 11th International Parallel Data Systems Workshop Archive Listing

Back to Full Workshop Archive Listing