CroCo: Cross-Modal Contrastive learning for localization of Earth Observation data

W.-H. Tseng, H.-Â. Lê, A. Boulch, S. Lefèvre and D. Tiede

International Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, ISPRS Congress, 2022

Paper Arxiv HAL

Abstract

It is of interest to localize a ground-based LiDAR point cloud on remote sensing imagery. In this work, we tackle a subtask of this problem, i.e. to map a digital elevation model (DEM) rasterized from aerial LiDAR point cloud on the aerial imagery. We proposed a contrastive learning-based method that trains on DEM and high-resolution optical imagery. We experimentally assess our framework on different data sampling strategies and hyperparameters. In the best scenario, the Top-1 score of 0.71 and Top-5 score of 0.81 are obtained. The proposed method is promising for feature learning from RGB and DEM for localization and is potentially applicable to other data sources too.