Workshop: AI on HPC: Performance Engineering, Challenges and Opportunities
Authors: Ajay Navilarekal Rajgopal (Leibniz Supercomputing Centre) and Nikolai Solmsdorf (Intel Corporation)
Abstract: Training large language models on production HPC systems requires parallelism configurations tailored to accelerator memory, interconnect topology, and software maturity. We present an empirical study of GPT-style training on Intel Data Center Max 1550 GPUs in SuperMUC-NG Phase 2, using the upstream Megatron-DeepSpeed stack without custom kernels, evaluating 3.6B-, 20B-, and 175B-parameter models to quantify how tensor parallelism, pipeline parallelism, micro-batch size, and gradient accumulation affect throughput and feasibility. Results show a pronounced penalty when tensor-parallel collectives cross the eight-tile node boundary, while more micro-batches amortize pipeline bubbles until throughput saturates. Bayesian optimization of the 175B space yields a configuration reaching 58.91 TFLOP/s per tile; scaling to 128 nodes (1024 tiles) retains 91% weak-scaling and 82% strong-scaling efficiency. We derive an empirical cost decomposition from pipeline depth, micro-batch size, and gradient accumulation that bounds what further auto-tuning alone can deliver on this system.
Back to AI on HPC: Performance Engineering, Challenges and Opportunities Archive Listing
Back to Full Workshop Archive Listing