Technical Papers Archive

BatchFlow: Benefit-Aware Data Pipeline Allocation and Batch Reuse for Multi-Job Training


Authors: Patrick Watters and Xiaolong Ma (University of Nevada, Reno); Syed Zawad (IBM Research); Jingyuan Zhang (George Mason University (GMU)); Yue Cheng (University of Virginia); Lei Yang (University of Nevada, Reno); and Feng Yan (University of Houston)

Abstract: Deep learning training is often limited by data pipeline throughput, reducing accelerator utilization and increasing training time and cost. This is particularly challenging in multi-tenant environments, where jobs compete for storage and CPU resources. We present BatchFlow, a data pipeline service that improves aggregate training throughput by dynamically allocating data pipeline workers across jobs. BatchFlow uses online profiling to estimate how quickly each job consumes mini-batches and how quickly workers can prepare them, then allocates workers to jobs where data supply is most likely to fall behind training demand. BatchFlow also caches prepared mini-batches for reuse across jobs, reducing repeated reads from slow storage and redundant CPU preprocessing. When cache capacity is limited, a benefit-aware policy determines which mini-batches to prefetch and retain. Across multi-job workloads, BatchFlow increases aggregate training throughput by up to 3.8× and cost efficiency by up to 2.1× relative to prior systems.


Presentation: file

Back to Technical Papers Archive Listing