Workshops Archive

Pelicanvil: A Web Platform for Discovering, Browsing, and Accessing Federated Datasets on HPC


Workshop: HUST-26: 13th International Workshop on HPC User Support Tools

Authors: Branden Turner (Texas A&M, Purdue University); Guangzhen Jin (Purdue University); Bithi De (Purdue University, SageSure); and Howard Zhong and Emma Turetsky (Morgridge Institute for Research)

Abstract: Pelican is a data-federation platform, and the Open Science Data Federation (OSDF) is a national federation built on it, letting researchers pull external datasets into HPC workflows without maintaining a local copy. Whether this is viable in practice remains unclear. A feasibility study benchmarking Pelican/OSDF against local and object storage on a production HPC cluster found performance workable but surfaced two barriers: no federation-wide catalog surfaces dataset metadata, search, and streamability up front, and whether a dataset streams per file or downloads depends on how it's packaged at the origin. In response, we built Pelicanvil, an infrastructure-agnostic web platform running inside a production cluster's Open OnDemand portal that gives researchers a single interface for discovering, browsing, and accessing federated datasets, through generated code snippets or direct download. This work offers HPC support teams a measured answer on Pelican's production readiness and a concrete response to the barriers found.


Back to HUST-26: 13th International Workshop on HPC User Support Tools Archive Listing

Back to Full Workshop Archive Listing