Workshops Archive

Dynamic Planning for Scalable Data Movement in Accelerated Query Engines


Workshop: PDSW’26: The 11th International Parallel Data Systems Workshop

Authors: Richard Zamora, Tom Augspurger, Peter Entschev, Mads Kristensen, Lawrence Mitchell, Matthew Murray, Niranda Perera, Vyas Ramasubramani, Matthew Roeschke, and Benjamin Zaitlen (NVIDIA)

Abstract: At scale, structured-data processing is often a data-movement problem. Joins and aggregations must choose whether to shuffle, broadcast, or reduce intermediate results, and the right choice can depend on data properties that are hard to predict prior to execution. In this paper, we introduce dynamic planning, a query-optimization method that lets operators choose data-movement strategies at runtime. We have implemented this approach in GPU Polars, where RapidsMPF actors observe table chunks and exchange metadata before selecting a strategy. Controlled microbenchmarks and a PDS-H Q9 case study confirm that dynamic planning makes sensible runtime decisions while preserving both stability and performance.


Back to PDSW’26: The 11th International Parallel Data Systems Workshop Archive Listing

Back to Full Workshop Archive Listing