Workshops Archive

Profiling and Optimization Opportunities for SmartNIC-Assisted MoE Inference


Workshop: Twelfth International Workshop on Heterogeneous High-Performance Reconfigurable Computing (H2RC 2026)

Authors: Shining Yang and Martin Herbordt (Boston University)

Abstract: Mixture-of-Experts (MoE) models improve capacity through sparse expert activation but introduce bursty all-to-all communication and rank-level imbalance during inference. We profile decode-stage Mixtral-8x7B inference on a 4-GPU vLLM data- and expert-parallel setup and observe up to 3.63 ms expert-compute rank skew. We propose a SmartNIC-assisted dispatch design with destination batching, expert-aware admission, blocked-chunk reordering, and credit-based dispatch. A deterministic C++ simulator shows that the full policy reduces average expert-flow-chunk latency by 17.1% and packet count by 74.9% on the profiled workload, with 8.6–18.9% latency reduction across nine routing and arrival workloads. We further implement a trace-driven FPGA/HLS prototype on an AMD/Xilinx U280. A 1,024-chunk board-level replay executes without runtime errors and exactly matches simulator chunk records, demonstrating implementability of the proposed scheduling kernel.


Back to Twelfth International Workshop on Heterogeneous High-Performance Reconfigurable Computing (H2RC 2026) Archive Listing

Back to Full Workshop Archive Listing