Authors: Keita Teranishi (Oak Ridge National Laboratory (ORNL)), Manish Motwani (Oregon State University), Jeffrey Young (Georgia Institute of Technology), Swaroop Pophale (Oak Ridge National Laboratory (ORNL)), Ignacio Laguna (Lawrence Livermore National Laboratory (LLNL))
Abstract: Effective software testing plays a critical role in guaranteeing the performance, correctness, and reproducibility of software. Testing high-performance computing (HPC) software poses unique challenges due to massive parallelism, concurrency and heterogeneity, the scale of target platforms, lack of oracles, and application-specific verification and verification. These challenges are now amplified by AI coding tools generating and translating HPC code faster than humans can review it. In this BoF, a panel of experts and the audience will discuss methodologies and challenges in HPC software testing, including verification of AI-generated code to deepen our understanding of this crucial aspect of HPC software development.
Long Description: This Birds of a Feather session is focused on engaging a wider audience in topics related to software testing, with a specific focus on testing and verification strategies for high-performance computing (HPC) software and applications.
Software testing plays a crucial role in ensuring the performance, correctness, and reproducibility of software. Robustness not only improves productivity, but also improves trust in the outcomes of simulations and analyses. While significant progress has been made in testing methodologies, techniques, and tools across software engineering domains, the adoption of these testing practices in high-performance scientific computing lags behind. This is due to unique requirements such as accurate floating-point arithmetic, massive parallelism, and concurrency in numerical algorithms, as well as heterogeneity in system architectures and the need for performance portability across node architectures and accelerators. Additional challenges arise from the lack of oracles, nondeterminism in program execution, and emerging floating-point data types driven by AI and machine learning, making conventional testing methodologies less feasible.
These challenges are compounded by a fundamental shift in how software is produced. Large language models and agentic AI coding tools are generating, translating, and refactoring code at unprecedented pace and scale, and HPC is no exception: AI assistants are increasingly used to port legacy scientific codes to new programming models, generate GPU kernels, and modernize decades-old codebases. Moreover, agentic AI and iterative AI-based code synthesis and debugging consume massive numbers of tokens, yet often yield only surface-level improvement: some generated codes pass compilation and a fixed set of unit or property tests, while deeper correctness issues, such as numerical accuracy, parallel semantics, and portability, remain unverified. The volume of code requiring validation is growing much faster than human review capacity, and AI-generated code can be subtly incorrect in ways that compile and run but silently produce wrong results. Consequently, many HPC test suites lack comprehensive coverage and a systematic, scalable approach to verification and validation.
This BoF will discuss the state of the art and the pitfalls and challenges that limit current testing strategies. Key topics include testing and verification of AI-generated and AI-translated code, oracle construction for numerical software, correctness under floating-point tolerance and nondeterminism, and scaling verification and validation to match the pace and cost of AI-assisted development. An ideal outcome would be agreement among the panel and audience on which issues to promote through classes, tutorials, workshops, or new research programs. We seek attendees' feedback on which techniques would most increase their adoption of testing best practices in the era of AI-assisted development.
The BoF will be an expert panel with pre-seeded questions, followed by extensive Q&A with attendees. Our confirmed speakers are Manish Motwani of Oregon State University, Swaroop Pophale of Oak Ridge National Laboratory, and Ignacio Laguna of Lawrence Livermore National Laboratory. All speakers are experienced practitioners of scientific software development and program correctness at scale. We will use online polling to gauge attendee interest, and outcomes will be shared online and through future workshops planned by the organizers.
Website: https://ssecenter.cc.gatech.edu/sc26-software-testing-bof/