Birds of a Feather Archive

Blended Infrastructure for Advanced Computational Workflows: From Accelerated Discovery to Agentic Automation


Authors: Felix Schürmann (Google LLC), Jay Boisseau (Google LLC), Ilkay Altintas De Callafon (University of California, San Diego (UCSD); San Diego Supercomputing Center (SDSC)), Costas Bekas (Citadel Securities), Fernanda Foertter (University of Alabama), Bob Sorensen (Hyperion Research), Rick Stevens (Argonne National Laboratory, University of Chicago)

Abstract: As HPC accelerates into an era of hybrid infrastructure and pervasive AI, computational research increasingly calls for optimizing blended on-prem and cloud infrastructure for advanced workflows that will accelerate future discoveries. This session begins with the drivers: cutting-edge science workflows challenging traditional infrastructure, and agentic AI workflows that autonomously drive the entire scientific loop from hypothesis generation to results analysis. We then review the current capabilities and gaps for managing and using blended HPC infrastructure as the backbone of these advanced workflows. Attendees will engage in discussion and interactive polling, collaboratively identifying gaps and opportunities for the community to address.

Long Description: The landscape of computational science is rapidly evolving, driven by increasingly complex, accelerated computational workflows alongside the emergence of autonomous, multi-agent AI research ecosystems. Today's cutting-edge science demands infrastructure capable of supporting massive data pipelines, high-throughput AI inference, and complex, coupled simulations. Concurrently, AI is advancing toward collaborative multi-agent harnesses designed to govern continuous, end-to-end research loops—autonomously formulating hypotheses, generating and optimizing simulation code, dispatching hybrid jobs, and evaluating multidimensional outputs. To interface with complex computing fabrics, these agent harnesses rely on "Skills"—standardized, dynamic tool adapters that connect AI agents directly to HPC schedulers, data-staging pipelines, and domain solvers. For these advanced workflows and autonomous agents to reliably steer scientific campaigns across distributed clusters, they require an underlying infrastructure that provides highly programmable, observable, and resilient execution environments backed by robust governance, non-human workload accounting, and human-in-the-loop oversight. Providing the critical backbone for these diverse and demanding workloads brings us to a deeply interconnected imperative: mastering the pragmatic management and effective usability of blended on-premise and cloud resources today. As computational workflows incorporate exponential data volumes, specialized AI accelerators, heterogeneous hardware, and multiple AI model providers, hybrid environments have shifted from an operational alternative to a production necessity. However, effectively integrating this blended infrastructure and making it seamlessly usable—with varying prioritizations for data placement, cost, and performance—remains a major challenge. Segregated allocations,

fractured workload scheduling, severe data-gravity bottlenecks, complex container portability, and unpredictable cloud-bursting economics routinely impede operational efficiency. Resolving these foundational infrastructure hurdles is the mandatory prerequisite for supporting the full spectrum of advanced science—from today's extreme-scale workloads to tomorrow's agentic workflows. Ultimately, uniting resilient hybrid infrastructure with next- generation computational workflows aims to transform global cyberinfrastructure into self- driving discovery engines, dramatically compressing the timeline from initial hypothesis to transformative scientific breakthrough. This session will establish a comprehensive bridge across these complementary technical layers: Top-Down Advanced Workflows and Agentic Ecosystems: We examine the emerging patterns of cutting-edge scientific and technical computing, spanning from data-intensive pipelines to multi-agent AI ecosystems. Discussion centers on the unique demands these workflows place on infrastructure, including the design and standardization of "Skills"— dynamic, plug-and-play tool adapters that enable AI agents to safely interact with distributed infrastructures, job schedulers, parallel file systems, and domain solvers. Furthermore, we will look at governance frameworks for highly automated and non-human workloads, focusing on provenance, reproducibility, auditability, and human-in-the-loop oversight. Bottom-Up Infrastructure Foundations: We explore state-of-the-art architectures for managing blended HPC and cloud resources for the evolving scientific loop. Key technical focal points include production-grade deployments of federated and hybrid Slurm schedulers, asynchronous data synchronization across tiered storage systems, containerized workload segregation (e.g., Apptainer, Kubernetes), and telemetry-driven frameworks for balancing computational throughput against cloud expenditure and software licensing constraints. By uniting blended infrastructure optimization with the forward-looking vision of advanced and agentic research workflows, this session delivers an essential roadmap for system administrators, research software engineers, and facility leaders seeking to optimize hybrid operations today while laying the groundwork for the future of accelerated, AI-driven discovery.


Back to Birds of a Feather Archive Listing