Workshop: AI4S: 8th Workshop on Artificial Intelligence and Machine Learning for Scientific Applications
Authors: Corey Adams, Peter Harrington, Akshay Subramaniam, Mohammad Shoaib Abbas, Jaideep Pathak, Dallas Foster, and Mike Pritchard (NVIDIA)
Abstract: Scientific Machine Learning (SciML) faces unique challenges for extreme-resolution data, with mitigations that often fail to scale or degrade the accuracy of trained models. Some specialized methods have achieved remarkable results in training models or performing inference on massive spatial datasets with bespoke techniques, but each is a model-specific effort; there is no general framework that makes domain parallelism accessible to SciML practitioners at large. We introduce \texttt{ShardTensor}, a PyTorch-native abstraction that brings domain parallelism into the standard training and inference workloads the way DDP brought data parallelism: non-invasively, without model changes, and composable with existing data- and model-parallel techniques. We describe the dispatch design that makes this possible, and show on benchmarks and production applications that unmodified models train beyond single-GPU memory, scale near-linearly at large input sizes, and run faster inference on high-resolution data.
Back to AI4S: 8th Workshop on Artificial Intelligence and Machine Learning for Scientific Applications Archive Listing
Back to Full Workshop Archive Listing