Research and ACM SRC Posters Archive

Transit-Layer Tensor Representation for Serverless Inference


Poster Type: ACM SRC Posters

Author: Ben Mechels (University of Minnesota, University of Chicago)

Abstract: The scale of frontier machine learning systems has led to an increasing need to efficiently store and transfer model weights. Existing methods for serverless inference employ scheduling policies to reduce cold start and model swapping latency, but have largely reached the limitations of hardware. In this paper, we introduce Condense, an ultra-fast method of compressing tensor weights across applications. We implement Condense in a serverless runtime as a transit-layer representation and evaluate the latency reduction. Across Llama models from 1B to 13B parameters, Condense reduces stored weight sizes by 2.27x-2.38x and improves cold-start loading time by 1.92x-2.17x over ServerlessLLM while retaining competitive downstream accuracy. We demonstrate that compact tensor representations can be generated quickly to provide significant runtime reductions across machine learning applications.

Best Poster Finalist (BP): no
Poster: PDF
Poster Summary: PDF


Back to Poster Archive Listing