Workshop: PDSW’26: The 11th International Parallel Data Systems Workshop
Authors: Xinxin Mei (Thomas Jefferson National Accelerator Facility (JLab)); Jie Ren (College of William & Mary); and Jon Zarling, Cameron Clarke, and Vardan Gyurjyan (Thomas Jefferson National Accelerator Facility (JLab))
Abstract: Scientific experiments are shifting from batched-file acquisition to online streaming readout, where O(1) MB detector frames flow directly to HPC facilities for real-time processing. Each HPC node runs a multi-stage workflow, and the slowest stage determines the overall throughput. On a pair of 100 Gbps nodes, throughput drops from 35 Gbps to 6 Gbps once downstream processing is enabled. Based on a 2024 experiment, we estimate that streaming live data would require about 300 Perlmutter nodes. This work explores Distributed Asynchronous Object Storage (DAOS) as a burst buffer that decouples the counting-house upstream from HPC downstream processing, such that fewer nodes can absorb the same traffic. Writing raw 8.6 KB packets to DAOS through an asynchronous queue achieved 32 Gbps end-to-end, limited by a single DAOS connection. Ongoing Aurora experiments suggest near-linear client-side scaling. We outline open challenges: small-object read penalties, tail latency under continuous production, and the throughput–latency trade-off.
Back to PDSW’26: The 11th International Parallel Data Systems Workshop Archive Listing
Back to Full Workshop Archive Listing