Workshops Archive

Decentralized Scheduling of Scientific Workflows


Workshop: 21st Workshop on Workflows in Support of Large-Scale Science (WORKS26)

Authors: Xavier Casas Moreno (Barcelona Supercomputing Center (BSC)); Komal Thareja and Anirban Mandal (Renaissance Computing Institute, University of North Carolina - Chapel Hill); Ewa Deelman (University of Southern California); and Rosa M. Badia and Francesc Lordan (Barcelona Supercomputing Center)

Abstract: Modern scientific workflows are evolving with the emergence of the Compute Continuum, which integrates geographically distributed, heterogeneous, and dynamic computing resources across HPC systems, cloud platforms, edge devices, and scientific facilities. In this context, traditional centralized Workflow Management Systems (WMSs) face scalability and fault-tolerance limitations, motivating the exploration of decentralized scheduling mechanisms.

In this paper, we present a general approach for implementing decentralized scheduling in WMSs, enabling distributed task submission, collaborative task allocation, and remote task execution. Using this approach, we implement a fully decentralized consensus-based scheduler in which agents collectively agree on task assignments without relying on a central coordinator or shared global state. The scheduler is integrated into the COMPSs WMS and evaluated on the MareNostrum 5 supercomputer using a distributed GridSearch application.

The results demonstrate improvements in workflow execution time and load balancing across agents. Our general approach provides a foundation for implementing decentralized scheduling mechanisms.


Back to 21st Workshop on Workflows in Support of Large-Scale Science (WORKS26) Archive Listing

Back to Full Workshop Archive Listing