Workshop: 13th Annual International Workshop on Innovating the Network for Data Intensive Science — INDIS
Authors: Manas Das (Southern Illinois University Edwardsville)
Abstract: Desk-side AI accelerator clusters enable local LLM execution, yet their networked system behavior remains uncharacterized. We evaluate a four-node NVIDIA GB10 (Grace Blackwell Architecture) cluster connected via a commodity Ethernet switch across twenty distributed inference configurations spanning dense and mixture-of-experts models. Four key findings emerge. First, decode is memory-bandwidth bound: single-node and pipeline-parallel setups achieve 0.69–0.86 of the memory-bandwidth roofline, while tensor parallelism divides per-node weight reads to accelerate decoding by up to 2.5x over RDMA, inverting standard datacenter trade-offs. Second, link bandwidth is not a bottleneck; per-node traffic consumes under 4% of the 200 Gb/s link without congestion. Third, communication latency remains critical, as RDMA over Converged Ethernet lowers end-to-end token latency by 2.0x to 7.1x compared to TCP. Finally, enabling expert parallelism alters inter-node traffic by less than 0.5%, reflecting vLLM’s default all-reduce backend rather than an intrinsic property of expert parallelism.
Back to 13th Annual International Workshop on Innovating the Network for Data Intensive Science — INDIS Archive Listing
Back to Full Workshop Archive Listing