Workshops Archive

Out-of-View Depth and Motion Prediction in Robotic Endoscopy


Workshop: Scale to See: Scalable AI for Scientific Imaging and Spatiotemporal Data

Authors: Md Siddique (Columbia University); Neil Getty (Argonne National Laboratory); and Leonardo Borgioli (University of Illinois Chicago, Argonne National Laboratory (ANL))

Abstract: The endoscope can observe only a narrow cone of the surgical field. Tissue leaving this cone deforms, which can lead to injuries if unnoticed. The hidden region has no visible pixels, so depth and motion must be predicted rather than measured. Reusing the last observed depth is a strong baseline over short intervals, but its error grows as tissue deforms, reaching 9.05 mm at the longest interval. We adopted a pretrained V-JEPA2-AC encoder with low-rank adaptation, where two decoders predict depth and optical flow for the hidden region from a reference RGB-D frame and a later view showing 16% of the frame. Under leave-one-organ-out cross-validation on four ex vivo DRENDS organs, with CRCD in every training fold, the adapted model reduces hidden-region depth RMSE from 6.852 to 6.197 mm, and endpoint error against RAFT pseudo-flow from 3.285 to 3.004 px relative to assuming zero motion.


Back to Scale to See: Scalable AI for Scientific Imaging and Spatiotemporal Data Archive Listing

Back to Full Workshop Archive Listing