Workshops Archive

From Pilots to Production: Lessons in Cross-Institutional Federated Training and AI for Science


Workshop: 4th Annual Trillion Parameter Workshop: Building Open AI Infrastructure, Models, and Agentic Systems for Science

Authors: Olivera Kotevska (Oak Ridge National Laboratory), Max Carlson (Sandia National Laboratories), Yan Gao (Flower Labs), Francis Jeanson (Ontario Brain Institute), Yijiang Li (Argonne National Laboratory), Mohammad Naseri (Flower Labs), Minseok Ryu (Arizona State University), Sahil Tyagi (Oak Ridge National Laboratory), Jerry Watkins (Sandia National Laboratories), Feiyi Wang (Oak Ridge National Laboratory), and Ravi Madduri and Kibaek Kim (Argonne National Laboratory)

Abstract: Many valuable scientific datasets cannot be centralized because they are proprietary, export-controlled, classified, or subject to data-sovereignty restrictions. Federated AI therefore provides essential infrastructure for training models across institutions while keeping data local. We synthesize lessons from U.S. Department of Energy national laboratories, industry deployments, and the open-source community on moving federated AI from pilots to dependable multi-site production. The analysis uses a five-domain readiness framework and two frontier-scale deployments: federated large language model training across three U.S. National Nuclear Security Administration laboratories using heterogeneous NVIDIA and AMD accelerators, and cross-facility fine-tuning spanning Aurora, Polaris, Frontier, and Perlmutter across Intel, NVIDIA, and AMD systems. We highlight systems challenges in memory efficiency, reliability, and synchronization; summarize best practices for privacy, security, and decentralized trust; and identify open problems in asynchronous federation, leakage auditing, verification standards, and harmonized data contracts that motivate an international open-science working group.


Back to 4th Annual Trillion Parameter Workshop: Building Open AI Infrastructure, Models, and Agentic Systems for Science Archive Listing

Back to Full Workshop Archive Listing