Authors: Botao Wu and Martin Kong (Ohio State University)
Abstract: Large-scale tensor networks are a critical class of computations widely used in domains such as computational chemistry, machine learning, and others. Owing to their memory footprint and complexity, their execution requires distributed-memory clusters and deeply efficient communication.
This work proposes an enhanced non-linear formulation that unifies previously proposed modeling and optimization schemes for DAGs of tensor operators. Our new modeling and formulation finds high-performing mappings for large tensor networks. Our key scientific contribution is a novel representation of the hyper-dimensional grid of processing elements. The new abstraction exponentially expands the search space, yielding higher performance and enhanced scalability. Furthermore, we reformulate the mapping to exploit optimization procedures in Google CP-SAT, leading to over 50x scalability improvement relative to prior work.
We evaluate our approach on various DAGs targeting CPU and multi-GPU clusters. Our approach achieves average speedups of 4.01x and 1.73x over the Cyclops Tensor Framework and NVIDIA cuTensorNet, respectively.
Presentation: file