{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/minkloc-lidar-and-monocular-image-fusion-for","title":"MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition","arxiv_id":"2104.05327","date":"2021-04-12","proceeding":null,"authors":["Jacek Komorowski","Monika Wysoczanska","Tomasz Trzcinski"],"abstract":"We introduce a discriminative multimodal descriptor based on a pair of sensor readings: a point cloud from a LiDAR and an image from an RGB camera. Our descriptor, named MinkLoc++, can be used for place recognition, re-localization and loop closure purposes in robotics or autonomous vehicles applications. We use late fusion approach, where each modality is processed separately and fused in the final part of the processing pipeline. The proposed method achieves state-of-the-art performance on standard place recognition benchmarks. We also identify dominating modality problem when training a multimodal descriptor. The problem manifests itself when the network focuses on a modality with a larger overfit to the training data. This drives the loss down during the training but leads to suboptimal performance on the evaluation set. In this work we describe how to detect and mitigate such risk when using a deep metric learning approach to train a multimodal neural network. Our code is publicly available on the project website: https://github.com/jac99/MinkLocMultimodal.","url_abs":"https://arxiv.org/abs/2104.05327v2","url_pdf":"https://arxiv.org/pdf/2104.05327v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"minkloc-lidar-and-monocular-image-fusion-for","repo_url":"https://github.com/jac99/MinkLocMultimodal","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-place-recognition","task_name":"3D Place Recognition"},{"task_slug":"autonomous-vehicles","task_name":"Autonomous Vehicles"},{"task_slug":"metric-learning","task_name":"Metric Learning"},{"task_slug":"multimodal-deep-learning","task_name":"Multimodal Deep Learning"},{"task_slug":null,"task_name":"Point Cloud Retrieval"},{"task_slug":"visual-place-recognition","task_name":"Visual Place Recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-place-recognition-on-cs-campus3d","task":"3D Place Recognition","dataset":"CS-Campus3D","model":"Minkloc3Dv2","rank_in_archive_order":3,"of":7,"metrics":{"AR@1":"67.06","AR@1 cross-source":"52.46","AR@1%":"76.68","AR@1% cross-source":"83.48"},"uses_additional_data":false},{"leaderboard":"/sota/visual-place-recognition-on-oxford-robotcar-1","task":"Visual Place Recognition","dataset":"Oxford RobotCar (LiDAR 4096 points+RGB)","model":"MinkLoc++ (LiDAR+RGB)","rank_in_archive_order":1,"of":3,"metrics":{"recall@top1":"96.7","recall@top1%":"99.1"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.05327","atlas_url":"https://app.syntology.ai/?focus=2104.05327","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.05327"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jac99/MinkLocMultimodal","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cf42d9b216f9481c","entry":"get_max_per_row","repo":"jac99/MinkLocMultimodal","repo_kind":"official","path":"models/loss.py","file_url":"https://github.com/jac99/MinkLocMultimodal/blob/HEAD/models/loss.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf42d9b216f9481c"}},{"code_sha256_prefix":"4bd72deb0a0b5897","entry":"get_min_per_row","repo":"jac99/MinkLocMultimodal","repo_kind":"official","path":"models/loss.py","file_url":"https://github.com/jac99/MinkLocMultimodal/blob/HEAD/models/loss.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4bd72deb0a0b5897"}},{"code_sha256_prefix":"ffc92c5bb8278ff1","entry":"tensor2img","repo":"jac99/MinkLocMultimodal","repo_kind":"official","path":"datasets/augmentation.py","file_url":"https://github.com/jac99/MinkLocMultimodal/blob/HEAD/datasets/augmentation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ffc92c5bb8278ff1"}},{"code_sha256_prefix":"a443d3b499a823b6","entry":"image4lidar","repo":"jac99/MinkLocMultimodal","repo_kind":"official","path":"datasets/oxford.py","file_url":"https://github.com/jac99/MinkLocMultimodal/blob/HEAD/datasets/oxford.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a443d3b499a823b6"}},{"code_sha256_prefix":"7cd8433380d56d84","entry":"ts_from_filename","repo":"jac99/MinkLocMultimodal","repo_kind":"official","path":"datasets/oxford.py","file_url":"https://github.com/jac99/MinkLocMultimodal/blob/HEAD/datasets/oxford.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cd8433380d56d84"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}