Technical Papers Archive

Exascale Distillation Workflows for Protein Structure Prediction on El Capitan


Authors: Nikoli Dryden, Tal Ben-Nun, Pier Fiedorowicz, and Tom Benson (Lawrence Livermore National Laboratory (LLNL)); Vinay S. Swamy and Colin Kalicki (Columbia University); Vinayak Gokhale, J. Austin Ellis, and Nicholas Malaya (Advanced Micro Devices (AMD)); Bronis R. de Supinski (Lawrence Livermore National Laboratory (LLNL)); Mohammed AlQuraishi (Columbia University); and Brian C. Van Essen (Lawrence Livermore National Laboratory (LLNL))

Abstract: Understanding the structure of proteins is critical to many biological applications. Recent deep learning approaches, such as AlphaFold, make high-quality structure predictions. However, training state-of-the-art models requires producing distillation datasets, which need several times more compute than training. We describe ElMerFold, an optimized distillation data generation workflow that scales to 11,200 nodes (44,800 APUs) on El Capitan, by leveraging a combination of inference, workflow, I/O, and resiliency innovations. We demonstrate this for the OpenFold 3 monomer protein distillation dataset, which uses a complex workflow centered on a set of pretrained OpenFold 2 models to predict structures for ~41 million proteins. ElMerFold achieves a 45.4x speedup over the state-of-the-art, predicting 6,718 structures/s, and achieving 1.99 Eflop/s peak for FP32 model inference (1.81 Eflop/s sustained including I/O). As distillation and synthetic data become more common, these workflows will help drive scientific advancement.



Back to Technical Papers Archive Listing