Birds of a Feather Archive

Trustworthy Data at Scale: HDF5 in the Age of AI-Driven Science


Authors: Michael Breitenfeld (The HDF Group), Elena Pourmal (LifeBoat, LLC), Suren Byna (The Ohio State University), Quincey Koziol (NVIDIA Corporation), Anthony Kougkas (Illinois Institute of Technology)

Abstract: HDF5 has anchored HPC I/O for 25 years, and AI-driven science is redefining what "trustworthy data at scale" demands: verifiable provenance, concurrent I/O, and cloud-native interoperability. Short talks from domain scientists launch interactive segments where an opt-in agentic survey captures attendee priorities, pain points, and open questions in real time, with aggregate results used to steer the discussion. This session is for seasoned HDF5 developers and domain scientists, and anyone just starting out or wondering what HDF5 is about: bring the problems you'd like to solve, and help steer the discussion and provide input to future HDF5 roadmap planning.

Long Description: Goals: AI is now a first-class HPC workload, and it is exposing a gap: models are only as trustworthy as the data beneath them. This BoF gathers the HDF5 community, developers, existing practitioners, experimentalists, and newcomers arriving through PyTorch and h5py, to align on what "trustworthy data at scale" must mean in the AI era, and to gather concrete requirements live, in the room, that feed directly into the HDF5 roadmap.

Topics: We anchor on three capabilities that matter for AI-era workloads and represent genuinely new HDF5 2.x work: verifiable data integrity and provenance, so a dataset can prove where it came from and that it hasn't changed, HDF5's durable advantage over cloud-native formats; concurrent, multithreaded chunked I/O for high-throughput data loading into training pipelines; and interoperability with cloud-native and other formats through the VOL connector architecture. Rather than survey these only through slides, we make them tangible through audience-driven prompts and a live requirements-gathering workflow: attendees respond to structured questions about HDF5 pain points, AI-era data needs, and trust requirements, and moderators use aggregate responses to shape the discussion. This opens a practical conversation about what AI workflows need from scientific data formats, and what evidence, provenance, and human oversight are required before AI-assisted data tools can be trusted.

Relevance to the HPC community: HDF5 is a core software component for every major HPC vendor, with over 1,000 GitHub projects depending on it, spanning national laboratories, industry (aerospace, energy, engineering simulation), academia, facilities and operations teams, and the open-source ecosystem. As ML's share of HPC cycles grows, its unconventional I/O patterns strain assumptions that classic HPC codes never tested. Every community that stores or trains on scientific data has a stake in getting trust, concurrency, and interoperability right.

Held before, and what changes: This BoF has run at SC14 through SC25, historically as a roadmap update followed by open Q&A. This year, five-minute domain talks replace longer technical presentations, and an opt-in agentic survey, collecting structured priorities and short explanations from attendees, replaces passive panels, with aggregate results published afterward as a community requirements snapshot. What stays the same: a roadmap update, community experts sharing real deployments, and an open floor for user feedback and pain points.

Expected outcomes include a live snapshot of community priorities, a ranked list of AI/ML pain points and requirements, candidate follow-on working groups, and documented input for HDF5 roadmap discussions. Organizations including The HDF Group, Lifeboat LLC, The Ohio State University, and NVIDIA convene this session, with the discussion agenda shaped in real time by attendee input.


Back to Birds of a Feather Archive Listing