Workshops Archive

Move Metadata, Not Data: A Register-in-Place Lakehouse for Large Autonomous-Driving Datasets


Workshop: PDSW’26: The 11th International Parallel Data Systems Workshop

Authors: Jey Kang, Kangryeol Kim, JinWang Mok, Ho Kim, Hyunje Sung, Sun Park, and JongWon Kim (Gwangju Institute of Science and Technology)

Abstract: Even when autonomous-driving data is available on shared storage, analysis pipelines often copy and reorganize it. To register sensor files without copying them, we built an Apache Iceberg lakehouse that records their locations and metadata. We registered about 647,000 lidar and radar Parquet files from a resident ∼13TB sample of NVIDIA’s 133 TB PhysicalAI Autonomous Vehicles dataset. Registering these Parquet files over NFS took approximately 2 h 8 min, based on log timestamps. In a separate four-scale experiment, registration time grew approximately linearly with file count. Timed after warm-up in that experiment, eleven queries over metadata or materialized tables had median latencies of 21–352 ms as total registered rows grew 36×; these timings do not characterize general sensor or media scans. Deployment also exposed an ownership hazard when DROP TABLE PURGE unintentionally deleted ∼11TB of source data. We describe safeguards and are developing a copy-based baseline.


Back to PDSW’26: The 11th International Parallel Data Systems Workshop Archive Listing

Back to Full Workshop Archive Listing