Birds of a Feather Archive

Data Readiness for AI in HPC: Building a Community Around Dimensions, Metrics, and Tools


Authors: Suren Byna (Ohio State University, Lawrence Berkeley National Laboratory (LBNL)), Jean Luca Bez (Lawrence Berkeley National Laboratory (LBNL)), Wes Brewer (Oak Ridge National Laboratory (ORNL)), Shreyas Cholia (Lawrence Berkeley National Laboratory (LBNL))

Abstract: AI now drives discovery across HPC science, yet models are only as trustworthy as the data they use. Many teams still lack a shared way to judge whether their data is ready for AI. This BoF brings together the HPC and scientific communities to define data readiness for AI using seven practical dimensions: quality, understandability, fair and unbiased representation, structure and organization, feature importance, governance covering ethics, security and privacy, and AI-specific needs such as labels and distributions. Through short talks and open discussion, we will compare practices, identify gaps, and begin building shared metrics, tools, and benchmarks.

Long Description: The value of any scientific AI model is limited by the readiness of the data used to build it. Today, there is no common practice or understanding of what makes data ready for AI, so groups assess readiness in inconsistent and ad hoc ways. This BoF brings together researchers, facility staff, and practitioners to discuss current practices, to build that shared understanding, and to grow a community around it.

We define AI-ready data as machine-readable, having undergone domain-appropriate cleaning, validation, and feature engineering, and contain metadata to support reproducible training and evaluation. We frame data readiness for AI through a comprehensive set of dimensions that together cover the questions a team should ask before training a model. Quality comes first: how complete and accurate the data is and whether it stays consistent, timely, and in distribution. The understandability and usability dimension asks whether the data is documented well enough for others to interpret and reuse. Representativeness asks whether the data is fair and free of bias, reflecting the population or phenomenon it is meant to capture. Structure and organization concern how the data is formatted, stored, and arranged for efficient access. Feature importance points to which variables carry a useful signal for a given task. Governance brings in ethics, security, and privacy. Finally, AI-specific requirements include the annotations, labels, class distributions, drift, etc. that supervised and generative methods rely on.

These dimensions matter to the SC community because HPC now sits at the center of large-scale AI for science. HPC facilities host enormous datasets from simulations, experiments, and observations, and these datasets feed training pipelines that demand careful preparation. Poor AI readiness of data wastes compute cycles, produces unreliable models, and undermines reproducibility and trustworthiness. Storage, I/O, metadata, and workflow systems all shape readiness, which makes this a natural topic for SC attendees who design and operate these systems.

Our goals for the session are to reach a rough agreement on the dimensions of AI readiness, to collect the practices and pain points that teams encounter in evaluating data readiness and resolving issues, and to identify where shared metrics, tools, and benchmarks are most needed. To collect the feedback of the attendees in the BoF and also to enhance the engagement, we will conduct a live survey, where we “collect attendee responses," “show the results," and “discuss the results and points of view” to discuss the readiness dimensions, pain points, shared metrics, tools, etc.

The expected outcome is an ongoing community focused on data readiness for AI. We aim to produce a shared summary of the dimensions and open challenges, a list of interested contributors across academia, research labs, and industry, and a plan for follow-up activities such as a working group, a shared benchmark suite, and future SC sessions. We also want to connect people who view data readiness from different angles, including domain scientists, data engineers, storage researchers, and those focused on ethics and privacy, so that future tools reflect the full range of needs across HPC.

Website: https://sites.google.com/lbl.gov/drai-bof-sc26



Back to Birds of a Feather Archive Listing