Authors: Bjoern Enders (Lawrence Berkeley National Laboratory (LBNL), National Energy Research Scientific Computing Center (NERSC)), Adam Thompson (NVIDIA Corporation), Alex Upton (Swiss National Supercomputing Centre (CSCS)), Rafael Ferreira da Silva (Oak Ridge National Laboratory (ORNL)), Eli Dart (Energy Sciences Network (ESnet), Lawrence Berkeley National Laboratory (LBNL)), Rajive Joshi (RTI), Rachana Ananthakrishnan (University of Chicago)
Abstract: Emerging scientific needs are driving a new class of workflows that require near real-time processing, urgent computing, and time-sensitive decision-making. The traditional "compute-store-analyze" paradigm, which relies on dumping massive datasets to file systems for post-processing, has become an unsustainable bottleneck. It introduces unacceptable latency, and often renders high-frequency data analysis practically impossible.
This BoF discusses this challenge by exploring the architectural shift from file-based workflows to high-performance streaming. In this new paradigm, storage is treated not as a buffer for post-processing, but as a sink for final scientific results, while analysis, visualization, and reduction occur in-transit or in-situ.
Long Description: As High-Performance Computing (HPC) enters the Exascale era, the scientific community faces a critical data paradox: our ability to generate data via high-fidelity simulations and next-generation instruments vastly outpaces our ability to store it. In many domains—from climate modeling and cosmology to experimental physics—simulation outputs and sensor inputs now frequently reach the Petabyte range per run. The traditional "compute-store-analyze" paradigm, which relies on dumping massive datasets to parallel file systems for post-processing, has become an unsustainable bottleneck. It introduces unacceptable latency, wastes energy on data movement, and often renders high-frequency data analysis practically inaccessible.
This architectural shift from file-based workflows to high-performance streaming is the reason for this BoF. In this new paradigm, storage is treated not as a buffer for post processing once simulations are completed, but as a sink for final scientific results, while analysis, visualization, and reduction occur in-transit or in-situ.
This BoF is particularly timely given the rapid convergence of HPC and Artificial Intelligence. Modern scientific workflows increasingly rely on AI to steer simulations, train surrogate models on-the-fly, and integrate Digital Twins with real-time data from the edge. These "live" interactions require data to flow continuously between compute nodes and inference engines, bypassing the latency of the file system entirely.
While a few streaming workflows have emerged recently, many HPC, data, or network user facilities are not set out to support these workflows out-of-gate, for policy, scheduler, or hardware reasons. However, these workflows are the cornerstone of modern scientific applications, as their stringent timing requirements benchmark the ultimate integration of HPC, data, and networks into seamless compute-in-the-loop workflows for experimental or observational user facilities.
This BoF is another step further to build a community to discuss the software stacks, protocols, and hardware architectures required to make streaming a first-class citizen in HPC. The BoF will move the conversation beyond ad-hoc, application-specific implementations toward standardized, interoperable frameworks.
This BoF is the 6th edition of its kind. It follows successful BoFs at SC24, ISC25, SC25, SCA26 and ISC26 as well as the workshop “Stream2HPC 2026” at ISC26. The preceding BoFs and workshops revealed strong interest in streaming workflows, particularly around integration challenges with existing HPC infrastructure and the need for standardized approaches across facilities. We submitted a workshop proposal called Stream2HPC and received 2.8 rating attesting to the relevance of the subject. As the workshop was not included into the program we are submitting this BoF instead to keep up the momentum with the US community.
SC26 is an ideal venue for this BoF as it brings the major HPC user facility providers together with scientific computing users who already run a realtime, streaming computing workflow or expect that need to arise in the near future. This provides an excellent opportunity to screen for current solutions, and discuss limitations in HPC, data, and network providers. At ISC26 we also started the process for a community whitepaper on the topic and we hope to give a status report here and invite more people to participate.
Website: https://workflows.community/bof/sc26-streaming