{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multipoint-cross-spectral-registration-of","title":"MultiPoint: Cross-spectral registration of thermal and optical aerial imagery","arxiv_id":null,"date":"2020-11-18","proceeding":"2020 Conference on Robot Learning 2020 11","authors":["Florian Achermann","Andrey Kolobov","Debadeepta Dey","Timo Hinzmann","Jen Jen Chung","Roland Siegwart","Nicholas Lawrance"],"abstract":"While optical cameras are ubiquitous in robotics, some robots can sense\r\nthe world in several sections of the electromagnetic spectrum simultaneously,\r\nwhich can extend their capabilities in fundamental ways. For instance, many\r\nfixed-wing UAVs carry both optical and thermal imaging cameras, potentially allowing them to detect temperature difference-induced atmospheric updrafts, map\r\ntheir locations, and adjust their flight path accordingly to increase their time aloft.\r\nA key step for unlocking the potential offered by multi-spectral data is generating\r\nconsistent, multi-spectral maps of the environment. In this work, we introduce\r\nMultiPoint, a novel data-driven method for generating interest points and associated descriptors for registering optical and thermal image pairs without knowledge\r\nof the relative camera viewpoints. Existing pixel-based alignment methods are accurate but too slow to work in near-real time, while feature-based methods such\r\nas SuperPoint are fast but produce poor-quality cross-spectral matches due to interest point instability in thermal images. MultiPoint capitalizes on the strengths\r\nof both approaches. An offline mutual information-based procedure is used to\r\nalign cross-spectral image pairs from a training set, which are then processed by\r\nour generalized multi-spectral homographic adaptation stage to generate highly\r\nrepeatable interest points that are invariant across viewpoint changes in both spectra. These are used to train a MultiPoint deep neural network by exposing this\r\nmodel to both same-spectrum and cross-spectral image pairs. This model is then\r\ndeployed for fast and accurate online interest point detection. We show that MultiPoint outperforms existing techniques for feature-based image alignment using\r\na dataset of real-world thermal-optical imagery captured by a UAV during flights\r\nin different conditions and release this dataset, the first of its kind.","url_abs":"https://proceedings.mlr.press/v155/achermann21a","url_pdf":"https://proceedings.mlr.press/v155/achermann21a/achermann21a.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multipoint-cross-spectral-registration-of","repo_url":"https://github.com/ethz-asl/multipoint","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"interest-point-detection","task_name":"Interest Point Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}