Workshop: PDSW’26: The 11th International Parallel Data Systems Workshop
Authors: Aiden Lambert, Aaron Jezghani, and Deepa Phanish (Georgia Institute of Technology)
Abstract: The rapid growth of scientific and AI workloads challenges HPC centers to efficiently move large datasets across high-performance storage systems. Data movement at this scale is constrained by I/O bandwidth, metadata processing, filesystem semantics, and resource consumption, while existing open-source and commercial solutions may lack required performance, resource guarantees, or cost effectiveness. Motivated by a large-scale storage migration at our HPC facility, we developed copy2, a resource-aware transfer tool for parallel and distributed storage systems including Lustre and VAST Data. copy2 combines fine-grained parallel metadata traversal, inode-based hard-link preservation, highly concurrent asynchronous I/O, and bounded memory management. Component-level and end-to-end evaluations across diverse file sizes and counts show 2-4x higher transfer bandwidth and 3-5x lower energy consumption than rclone and fpsync, while adhering to user-defined memory bounds. Production deployments involving hundreds of millions of files further demonstrate its effectiveness for predictable, high-throughput scientific data movement at scale.
Back to PDSW’26: The 11th International Parallel Data Systems Workshop Archive Listing
Back to Full Workshop Archive Listing