Workshop: 8th International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)
Authors: Richard Henwood, John Lockman, and Luke Wilson (Dell Technologies)
Abstract: Large-scale compute (HPC) systems often deploy InfiniBand (IB) networks for high-bandwidth and low latency communication. In an HPC system, a TCP network is typically used for control-plane and general-purpose communication. The last decade has seen the rise of cloud-native software patterns which are characterized by container based software distribution and communication over TCP/IP.
This paper introduces the Bipartible Architecture Pattern that transparently redirects HTTP traffic onto IB without changing workload implementation or network configuration. This is achieved by introducing a service on the host(s) that intercepts HTTP requests and transforms them into global filesystem requests. This maintains the HTTP interface while leveraging the IB network for data transfer.
The feasibility of this pattern is demonstrated by implementing a prototype for Garage (S3) and Pulp (repository). We offer this pattern as one of many alternative approaches to address the performance challenges of deploying scientific software containers at scale on HPC environments.
Back to 8th International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC) Archive Listing
Back to Full Workshop Archive Listing