Technical Papers Archive

MPCAR: Scalable and Secure Multi-Party Computation-Based AllReduce


Authors: Hao Qi (University of Florida); Weicong Chen (University of California, Merced); Chenghong Wang (Indiana University); Caiwen Ding (University of Minnesota); and Xiaoyi Lu (University of Florida)

Abstract: Scalable and secure collective communication is critical in modern HPC and AI workloads, especially AllReduce. State-of-the-art (SOTA) solutions use homomorphic encryption (HE) to ensure data privacy, but incur complex key management and overheads, and weak integration with collective algorithms. We present MPCAR, the first multi-party computation-based AllReduce framework, which decouples AllReduce into two phases, data reduction and distribution, and applies tailored secure methods to each. MPCAR employs secret sharing MPC schemes for the data reduction phase and asymmetric one-time-padding-based protection for the distribution phase, while also allowing user-defined secure communication protocols. We co-design these schemes with multiple AllReduce algorithms and implement several system-level optimizations. Evaluated across five platforms spanning AWS and HPC clusters at scale, including two exascale supercomputers and scaling to 8,192 ranks, MPCAR improves latency/throughput by up to 30.5%/41.2%. MPCAR also outperforms SOTA by up to 21.3% on real-world workloads, including DNN applications and three DeepSeek-distilled models.


Presentation: file

Back to Technical Papers Archive Listing