Workshops Archive

Can LLMs Write Task-Based Parallel Code?


Workshop: 4th Annual Trillion Parameter Workshop: Building Open AI Infrastructure, Models, and Agentic Systems for Science

Authors: Eduardo Iraola and Néstor Escolà (Barcelona Supercomputing Center), Francesc Lordan (Barcelona Supercomputing Center (BSC)), and Rosa M. Badia (Barcelona Supercomputing Center)

Abstract: Large Language Models are increasingly integrated into software development, yet HPC remains demanding for them. Their performance has been measured for low-level, kernel-based parallel programming paradigms, but task-based and workflow orchestration paradigms, despite growing importance, have seen less rigorous evaluation. We extend the ParEval benchmark, originally demonstrated on C++ kernel-level models, to task-based paradigms and demonstrate it on PyCOMPSs, asking whether LLMs generate correct and efficient task-based parallel code. We further propose a multi-tier prompting scheme, from programming-model knowledge to infrastructure-aware coding, and evaluate the first two levels. Across 21 models, correctness and efficiency decouple: the most correct model is among the least efficient, and only one frontier model is strong in both. Open-weight models close the gap to the previous commercial generation with framework-specific guidance. The main barrier is framework knowledge, not the parallel algorithm, and correct code still achieves limited efficiency at the studied problem size.


Back to 4th Annual Trillion Parameter Workshop: Building Open AI Infrastructure, Models, and Agentic Systems for Science Archive Listing

Back to Full Workshop Archive Listing