Workshop: AI on HPC: Performance Engineering, Challenges and Opportunities
Authors: Hongyu Liu (Chalmers University of Technology and University of Gothenburg), Minyu Cui (Linnaeus University), and Miquel Pericàs (Chalmers University of Technology and University of Gothenburg)
Abstract: Fine-grained computation-communication overlap lets distributed Mixture-of-Experts (MoE) inference start communication on partial results. Compute and communication cooperative thread arrays (CTAs), however, contend for finite streaming-multiprocessor (SM) residency and progress in discrete waves, and a fixed partition cannot adapt to workload or kernel changes. We present a wave-quantized analytical model and launch-time resource manager that selects the communication-CTA count from routed-tile counts, kernel occupancy, residency constraints, and split-level readiness dependencies before launch, without profiling or recompilation. We integrate it into the COMET A100 implementation and evaluate three MoE models on four A100 GPUs under three parallelism configurations at the GEMM2+GatherRS operator, complete post-router MoE layer, and complete-model prefill levels. Speedups over COMET reach 4.218×, 2.584×, and 1.439× at these levels, respectively. At every feasible TP=2/EP=2 sequence length of at least 4096, our implementation outperforms Megatron core-TE and FastMoE TP+NCCL.
Back to AI on HPC: Performance Engineering, Challenges and Opportunities Archive Listing
Back to Full Workshop Archive Listing