Workshop: 21st Workshop on Workflows in Support of Large-Scale Science (WORKS26)
Authors: Nicolò Giacomini, Débora Pina, Rosa M. Badia, and Raül Sirvent (Barcelona Supercomputing Center)
Abstract: Accurately estimating execution times is essential in HPC, where users must request resources and wall-clock limits before execution. However, execution information relevant to prediction is typically scattered across profilers, schedulers, and workflow engines using incompatible representations. We argue that execution data should instead be treated as a standardized, reusable artifact of workflow execution rather than a disposable input to a single predictor. We present an architecture that captures distributed resource usage, packages it as Workflow Run RO-Crates, and persists it in a queryable hybrid storage architecture (ProvStor), enabling any execution environment producing compatible metadata to contribute to and reuse a shared knowledge base. We showcase an enactment of this architecture using the COMPSs workflow management system, introducing a SLURM-aware profiler, and show its value through a hybrid prediction methodology that combines the analytical Universal Scalability Law with Machine Learning models.
Back to 21st Workshop on Workflows in Support of Large-Scale Science (WORKS26) Archive Listing
Back to Full Workshop Archive Listing