Workshop: 21st Workshop on Workflows in Support of Large-Scale Science (WORKS26)
Authors: Nagarjuna Kandimalla, Bhanu Prakash Vangala, and Tanu Malik (University of missouri columbia)
Abstract: Distributed scientific workflow tasks exhibit signif- icant variation in memory consumption, making accurate peak memory estimation critical for resource management and avoid- ing out-of-memory (OOM) failures. Existing workload schedulers rely on static features like file sizes or tool names, which fail to capture non-linear memory scaling. We present Consumption- Aware Memory Prediction (CAMP), a closed-loop framework that utilizes fine-grained, consumption-level provenance—the exact number of bytes a task reads—measured dynamically via eBPF kernel probes. To minimize tracing overhead across parallel task executions at scale, we develop an active-learning-based, multi- gate selective auditing policy that samples only the most informa- tive workflow tasks for model retraining. We evaluate CAMP on thousands of production-scale bioinformatics workflow records executed under Slurm. Compared to size-based scheduling base- lines, incorporating dynamic data-consumption features reduces prediction median absolute percentage error by up to 40.31%.
Back to 21st Workshop on Workflows in Support of Large-Scale Science (WORKS26) Archive Listing
Back to Full Workshop Archive Listing