Workshops Archive

Obleth: Fair Multi-Tenant LLM Serving for Self-Hosted Campus Infrastructure


Workshop: HPC Systems Professionals Workshop (HPCSYSPROS26)

Authors: Johnathan Lee (Arizona State University)

Abstract: LLMs are becoming more popular, and costs are continuing to rise. Many universities are looking to host LLMs on local infrastructure to meet compliance, cost, and data-sovereignty requirements. Most AI gateways assume a cloud-provider backend, one that handles seemingly unlimited connections and natively supports capabilities that many self-hosted models lack. This is why I created Obleth, a source-available AI gateway designed for locally hosted infrastructure. Obleth's standout features include: a weighted fairshare admission scheduler dividing inference capacity proportionally across tenants; a Slurm cluster provisioner for opportunistic compute on preemptible partitions with automatic preemption recovery; an auto router that selects the best available model from a registered fleet based on capability, capacity, and cost; runtime-configurable capability extensions (boons) for things like on-premise content policy enforcement and MCP tool integration; and per-request pipeline tracing requiring no external telemetry. We evaluate fairness guarantees and gateway overhead with an included reproducible benchmark harness.


Back to HPC Systems Professionals Workshop (HPCSYSPROS26) Archive Listing

Back to Full Workshop Archive Listing