Workshops Archive

An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc


Workshop: AI4S: 8th Workshop on Artificial Intelligence and Machine Learning for Scientific Applications

Authors: Hong Zhang, Barry Smith, Satish Balay, Le Chen, Murat Keceli, Lois Curfman McInnes, and Junchao Zhang (Argonne National Laboratory)

Abstract: While LLMs have accelerated scientific code gen- eration, comprehensively evaluating generated code remains challenging. Many benchmarks emphasize functional correctness or task completion, which is insufficient for code built on pro- duction HPC libraries, where solver selection, API conventions, memory management, parallel awareness, and performance also matter. We introduce PETSCAGENT-BENCH, a multidimensional benchmark and agent-based framework for assessing whether AI-generated scientific code uses a production HPC library as an expert would. A tool-augmented evaluator compiles, executes, and measures code and combines deterministic checks with LLM-based assessments in a 14-evaluator pipeline spanning five categories: correctness, performance, code quality, algorithmic appropriateness, and library-specific conventions. A2A and MCP enable black-box evaluation of compatible coding agents. Across realistic PETSc problems, frontier models generate readable, well-structured code but struggle with correctness on challeng- ing problems and with library-specific conventions even when code compiles and runs—limitations that conventional pass/fail evaluation does not capture.


Back to AI4S: 8th Workshop on Artificial Intelligence and Machine Learning for Scientific Applications Archive Listing

Back to Full Workshop Archive Listing