Technical Papers Archive

4D Parallelism Unlocks Exascale Bayesian Neural Networks for High-Fidelity Atmospheric Modeling


Authors: Deifilia Kieckhefen, Juan Pedro Gutiérrez Hermosillo Muriedas, and Lars Helge Heyen (Karlsruhe Institute of Technology (KIT)); Mathis Bode (Forschungzentrum Juelich); Iida Hakulinen (CSC - IT Center for Science); Andreas Herten and Chelsea Maria John (Forschungzentrum Juelich); Thorsten Kurth (NVIDIA Corporation); Anni Moisala (CSC - IT Center for Science); Asena Karolin Özdemir and Kaleb Phipps (Karlsruhe Institute of Technology (KIT)); Oskar Taubert (CSC - IT Center for Science); and Arvid Weyrauch, Markus Götz, and Charlotte Debus (Karlsruhe Institute of Technology (KIT))

Abstract: We present BEAST, the first-ever Bayesian Swin Transformer for atmospheric forecasting on 0.25° global resolution. To overcome the associated computational bottlenecks, we devise an orthogonal 4D-parallelization scheme that introduces a unique domain-tensor-parallelism strategy and a novel uncertainty parallel method, enabling us to fully leverage GPU capacity and efficiently scale model training. For a 2.4-billion-parameter model, we achieve a peak performance of 3.96 EFLOP/s on 20,480 NVIDIA GH200 GPUs on the JUPITER supercomputer.

We train BEAST as a 700-million-parameter model with 96 random weight samples on 384 nodes on 40 years of data for nearly one million gradient updates. This model achieves predictive skill scores competitive with state-of-the-art probabilistic atmospheric AI models and numerical models, and can predict extreme events with exceptional skill, while generating large ensembles 3–4 times faster than the current-best AI model. Our contribution unlocks the potential of high-fidelity uncertainty quantification in atmospheric AI models.




Back to Technical Papers Archive Listing