Workshops Archive

Python-Based Distributed Training on AMD GPU Supercomputers: Lessons from Data, Pipeline, and Tensor Parallelism on LUMI


Workshop: High Performance Python for Science at Scale

Authors: Vijeta Sharma (RISE Research Institutes of Sweden)

Abstract: The growing computational demands of foundation AI models have made distributed training essential on modern supercomputers. This work presents an empirical study of data, pipeline, and tensor parallelism for Vision Transformer training on the LUMI supercomputer, an AMD GPU-based leadership-class HPC system. Using Python-based frameworks including PyTorch, FairScale, TensorFlow DTensor, and Colossal-AI within containerized ROCm environments, we evaluate framework compatibility, portability, implementation complexity, and reproducibility. Our findings show that software ecosystem readiness significantly impacts scalable AI training. PyTorch DDP and TensorFlow DTensor demonstrate reliable execution, while several CUDA-dependent approaches face deployment challenges on AMD architectures.


Back to High Performance Python for Science at Scale Archive Listing

Back to Full Workshop Archive Listing