Workshop: HPC Systems Professionals Workshop (HPCSYSPROS26)
Authors: Deepa Phanish, Aiden Lambert, Aaron Jezghani, Ruben Lara, Rachel Lombardi, Kenneth Suda, and Grigori Yourganov (Georgia Institute of Technology)
Abstract: Large-scale storage migrations are among the most challenging operational activities in high-performance computing (HPC) environments due to the need to preserve workflow continuity while moving petabyte-scale datasets across storage platforms. This paper presents the planning and execution of a live migration of 900 research projects totaling 3.28 PB of data across 3.30 billion files from Lustre-based storage to a VAST Data filesystem. We implemented a technology-agnostic namespace using automount abstractions and dual datapaths to decouple workflow migration from data movement. Supported by a custom toolchain for metadata-preservation, our pipeline integrated project prioritization, pre-copy transfers, incremental synchronization, and automated validation. Despite being a fully-managed migration, this framework minimized administrative overhead, preserved filesystem integrity, and achieved zero user-reported tickets for workflow disruption.
Back to HPC Systems Professionals Workshop (HPCSYSPROS26) Archive Listing
Back to Full Workshop Archive Listing