Authors: Tom St. John (Gimlet Labs), Murali Emani (Argonne National Laboratory), Miro Hodak (Advanced Micro Devices (AMD)), Tushar Krishna (Georgia Tech, InfraVana), Wolfgang De Salvador (Microsoft Corporation), David Kanter (MLCommons)
Abstract: As AI workloads scale exponentially, evaluating machine learning infrastructure requires rigorous, standardized benchmarking. This Birds of a Feather session brings together HPC researchers, hardware architects, and software engineers to discuss the future of the MLPerf benchmark suite. The discussion will address critical scaling challenges, heterogeneous accelerator evaluation, and performance metrics for disaggregated infrastructure. Participants will collaborate on driving reproducibility and expanding benchmarks to encompass emerging, complex AI workloads. Join us to share deployment insights and directly shape the benchmarking roadmaps guiding the next generation of supercomputing systems.
Long Description: The landscape of artificial intelligence is undergoing a fundamental shift. As machine learning models scale in size and complexity, the supercomputing community is moving away from traditional monolithic clusters toward heterogeneous computing environments and disaggregated infrastructure. While MLPerf has established itself as the industry standard for measuring ML performance, this architectural transition introduces unprecedented challenges in evaluation, workload orchestration, and cost-aware optimization. Over the past year, the MLPerf benchmark suite has evolved aggressively to keep pace with the explosion of complex, sparse, and agentic AI workloads.
This Birds of a Feather session provides an interactive forum for HPC researchers, system architects, and ML engineers to tackle these emerging evaluation bottlenecks emerging at the frontier of supercomputing. A primary focus of our discussion will be the rigorous benchmarking of heterogeneous accelerators. We will explore the complexities of measuring performance across diverse hardware ecosystems, addressing how underlying compiler toolchains, tensor algebra frameworks, and intermediate representations can drastically impact benchmark outcomes.
As the industry advances beyond static inference and training toward dynamic, multi-modal, and agentic AI workloads, our evaluation strategies must adapt accordingly. Traditional throughput metrics often fail to capture the systemic nuances of agentic orchestration, heterogeneous resource scheduling, and distributed memory management. MLPerf has recently incorporated its first agentic inference benchmarks leveraging Kimi K2.6 and Qwen3.6-35B-A3B models, measuring closed-loop, multi-turn trajectories for coding and enterprise workflows, rather than independent query rates. Participants will explore the complex serving bottlenecks these workloads introduce, such as continuous context growth, critical KV-cache reuse, and strict turn dependencies. We will also examine MLPerf’s new latency-constrained interactive reasoning benchmarks using DeepSeek-R1 and GPT-OSS-120B models.
The session will begin with brief opening remarks outlining the current state of the MLPerf Training and Inference suites, immediately transitioning into a lively, audience-driven town hall. Attendees are encouraged to share deployment insights, scaling frustrations, and optimization strategies. By convening experts in systems-level ML optimization, we aim to synthesize actionable feedback for the MLCommons community, ultimately helping to shape a benchmarking roadmap that reflects the real-world realities of next-generation supercomputing.
Website: https://mlcommons.org