Workshop: 4th Annual Trillion Parameter Workshop: Building Open AI Infrastructure, Models, and Agentic Systems for Science
Authors: Sam Foreman, Nathan Nichols, Varuni Sastry, Sam Wheeler, Khalid Hossain, Huihuo Zheng, Murali Emani, Filippo Simini, Marieme Ngom, Ethan Wong, and Venkatram Vishwanath (Argonne National Laboratory)
Abstract: AuroraGPT pre-trains open foundation models for science on Aurora, an HPE Cray EX system of ~127,000 Intel Max-series XPUs and one of the largest non-NVIDIA GPU systems in production. Training at this scale on a SYCL/oneCCL stack, rather than CUDA/NCCL, surfaces portability and reliability problems the usual toolchain does not address. We present an open, portable stack: ezpz, a launcher that runs the same unmodified PyTorch script across Intel, NVIDIA, and AMD, and an experiments/ezpz fork of TorchTitan. The completed AuroraGPT-2B run reached 7.77T tokens; on the migrated stack, an in-flight 20B run matches that fully-trained 2B's peak evaluation scores at comparable compute while consuming ~17x fewer tokens. We detail the failures behind these numbers: a silent loss-reporting bug caught only by bit-exact regression, an 80B instability that NaN'd three optimizers, and a launcher-native auto-retry protocol that eliminated a silent-failure mode responsible for nine lost runs in two weeks.
Back to 4th Annual Trillion Parameter Workshop: Building Open AI Infrastructure, Models, and Agentic Systems for Science Archive Listing
Back to Full Workshop Archive Listing