Authors: Felix Schürmann (Google LLC), Jay Boisseau (Google LLC), Ilkay Altintas De Callafon (University of California, San Diego (UCSD); San Diego Supercomputing Center (SDSC)), Costas Bekas (Citadel Securities), Fernanda Foertter (University of Alabama), Bob Sorensen (Hyperion Research), Rick Stevens (Argonne National Laboratory, University of Chicago)
Abstract: As HPC accelerates into an era of hybrid infrastructure and pervasive AI, computational research increasingly calls for optimizing blended on-prem and cloud infrastructure for advanced workflows that will accelerate future discoveries. This session begins with the drivers: cutting-edge science workflows challenging traditional infrastructure, and agentic AI workflows that autonomously drive the entire scientific loop from hypothesis generation to results analysis. We then review the current capabilities and gaps for managing and using blended HPC infrastructure as the backbone of these advanced workflows. Attendees will engage in discussion and interactive polling, collaboratively identifying gaps and opportunities for the community to address.
Long Description: The landscape of computational science is rapidly evolving, driven by increasingly complex,
accelerated computational workflows alongside the emergence of autonomous, multi-agent
AI research ecosystems. Today's cutting-edge science demands infrastructure capable of
supporting massive data pipelines, high-throughput AI inference, and complex, coupled
simulations. Concurrently, AI is advancing toward collaborative multi-agent harnesses
designed to govern continuous, end-to-end research loops—autonomously formulating
hypotheses, generating and optimizing simulation code, dispatching hybrid jobs, and
evaluating multidimensional outputs. To interface with complex computing fabrics, these
agent harnesses rely on "Skills"—standardized, dynamic tool adapters that connect AI
agents directly to HPC schedulers, data-staging pipelines, and domain solvers. For these
advanced workflows and autonomous agents to reliably steer scientific campaigns across
distributed clusters, they require an underlying infrastructure that provides highly
programmable, observable, and resilient execution environments backed by robust
governance, non-human workload accounting, and human-in-the-loop oversight.
Providing the critical backbone for these diverse and demanding workloads brings us to a
deeply interconnected imperative: mastering the pragmatic management and effective
usability of blended on-premise and cloud resources today. As computational workflows
incorporate exponential data volumes, specialized AI accelerators, heterogeneous hardware,
and multiple AI model providers, hybrid environments have shifted from an operational
alternative to a production necessity. However, effectively integrating this blended
infrastructure and making it seamlessly usable—with varying prioritizations for data
placement, cost, and performance—remains a major challenge. Segregated allocations,
fractured workload scheduling, severe data-gravity bottlenecks, complex container
portability, and unpredictable cloud-bursting economics routinely impede operational
efficiency.
Resolving these foundational infrastructure hurdles is the mandatory prerequisite for
supporting the full spectrum of advanced science—from today's extreme-scale workloads to
tomorrow's agentic workflows. Ultimately, uniting resilient hybrid infrastructure with next-
generation computational workflows aims to transform global cyberinfrastructure into self-
driving discovery engines, dramatically compressing the timeline from initial hypothesis to
transformative scientific breakthrough.
This session will establish a comprehensive bridge across these complementary technical
layers:
Top-Down Advanced Workflows and Agentic Ecosystems: We examine the emerging
patterns of cutting-edge scientific and technical computing, spanning from data-intensive
pipelines to multi-agent AI ecosystems. Discussion centers on the unique demands these
workflows place on infrastructure, including the design and standardization of "Skills"—
dynamic, plug-and-play tool adapters that enable AI agents to safely interact with distributed
infrastructures, job schedulers, parallel file systems, and domain solvers. Furthermore, we
will look at governance frameworks for highly automated and non-human workloads,
focusing on provenance, reproducibility, auditability, and human-in-the-loop oversight.
Bottom-Up Infrastructure Foundations: We explore state-of-the-art architectures for
managing blended HPC and cloud resources for the evolving scientific loop. Key technical
focal points include production-grade deployments of federated and hybrid Slurm
schedulers, asynchronous data synchronization across tiered storage systems,
containerized workload segregation (e.g., Apptainer, Kubernetes), and telemetry-driven
frameworks for balancing computational throughput against cloud expenditure and software
licensing constraints.
By uniting blended infrastructure optimization with the forward-looking vision of advanced
and agentic research workflows, this session delivers an essential roadmap for system
administrators, research software engineers, and facility leaders seeking to optimize hybrid
operations today while laying the groundwork for the future of accelerated, AI-driven
discovery.