Workshops Archive

Operationalizing Workflow Support on Slurm: A Site Toolkit for Choosing Nextflow Deployment Strategie


Workshop: HPC Systems Professionals Workshop (HPCSYSPROS26)

Authors: Nil Tianchen Mu (Arizona State University)

Abstract: Workflow-manager deployment choices determine which Slurm objects exist, how much controller load a workflow creates, and where accounting, policy, and failure isolation apply. We present an operator-run toolkit that freezes site and workload state, executes clean-start trials, attributes RPC demand to the benchmark identity, validates scientific completion, audits the campaign, and produces offline analyses. We apply it in three replicates to native Slurm, job arrays, HyperQueue, and Flux using a real 4,823-task Nextflow alignment workload on ASU’s Phoenix production cluster, with a private Dev cluster providing co-tenancy-free context. On Phoenix, arrays reduced both walltime and attributable RPC demand in every paired comparison while retaining per-element Slurm control. Flux reduced RPC demand further but ran longer than arrays; HyperQueue’s fastest median was not stable across replicates or clusters. We provide a constraint-first decision procedure, deployment configurations, and operational guidance for selecting a defensible site default.


Back to HPC Systems Professionals Workshop (HPCSYSPROS26) Archive Listing

Back to Full Workshop Archive Listing