{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/freereg-image-to-point-cloud-registration","title":"FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators","arxiv_id":"2310.03420","date":"2023-10-05","proceeding":null,"authors":["Haiping Wang","YuAn Liu","Bing Wang","Yujing Sun","Zhen Dong","Wenping Wang","Bisheng Yang"],"abstract":"Matching cross-modality features between images and point clouds is a fundamental problem for image-to-point cloud registration. However, due to the modality difference between images and points, it is difficult to learn robust and discriminative cross-modality features by existing metric learning methods for feature matching. Instead of applying metric learning on cross-modality data, we propose to unify the modality between images and point clouds by pretrained large-scale models first, and then establish robust correspondence within the same modality. We show that the intermediate features, called diffusion features, extracted by depth-to-image diffusion models are semantically consistent between images and point clouds, which enables the building of coarse but robust cross-modality correspondences. We further extract geometric features on depth maps produced by the monocular depth estimator. By matching such geometric features, we significantly improve the accuracy of the coarse correspondences produced by diffusion features. Extensive experiments demonstrate that without any task-specific training, direct utilization of both features produces accurate image-to-point cloud registration. On three public indoor and outdoor benchmarks, the proposed method averagely achieves a 20.6 percent improvement in Inlier Ratio, a three-fold higher Inlier Number, and a 48.6 percent improvement in Registration Recall than existing state-of-the-arts.","url_abs":"https://arxiv.org/abs/2310.03420v2","url_pdf":"https://arxiv.org/pdf/2310.03420v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"freereg-image-to-point-cloud-registration","repo_url":"https://github.com/WHU-USI3DV/FreeReg","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-to-point-cloud-registration","task_name":"Image to Point Cloud Registration"},{"task_slug":"metric-learning","task_name":"Metric Learning"},{"task_slug":"point-cloud-registration","task_name":"Point Cloud Registration"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.03420","atlas_url":"https://app.syntology.ai/?focus=2310.03420","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.03420"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/WHU-USI3DV/FreeReg","reach":{"status":"ok"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"f453003e6a80efab","entry":"dist2color","repo":"WHU-USI3DV/FreeReg","repo_kind":"official","path":"utils/drawer.py","file_url":"https://github.com/WHU-USI3DV/FreeReg/blob/HEAD/utils/drawer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f453003e6a80efab"}},{"code_sha256_prefix":"ab5e2aa28efbe74b","entry":"hpoints_to_points","repo":"WHU-USI3DV/FreeReg","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/WHU-USI3DV/FreeReg/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab5e2aa28efbe74b"}},{"code_sha256_prefix":"776fbdb9533c368d","entry":"make_matching_plot_fast","repo":"WHU-USI3DV/FreeReg","repo_kind":"official","path":"utils/drawer.py","file_url":"https://github.com/WHU-USI3DV/FreeReg/blob/HEAD/utils/drawer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"776fbdb9533c368d"}},{"code_sha256_prefix":"98e77c345bd91f49","entry":"matrix_from_quaternion","repo":"WHU-USI3DV/FreeReg","repo_kind":"official","path":"utils/r_eval.py","file_url":"https://github.com/WHU-USI3DV/FreeReg/blob/HEAD/utils/r_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98e77c345bd91f49"}},{"code_sha256_prefix":"f693a93c43e492f3","entry":"points_to_hpoints","repo":"WHU-USI3DV/FreeReg","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/WHU-USI3DV/FreeReg/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f693a93c43e492f3"}},{"code_sha256_prefix":"00f643de599beccf","entry":"quaternion_from_matrix","repo":"WHU-USI3DV/FreeReg","repo_kind":"official","path":"utils/r_eval.py","file_url":"https://github.com/WHU-USI3DV/FreeReg/blob/HEAD/utils/r_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00f643de599beccf"}},{"code_sha256_prefix":"2ca32b40b0e54814","entry":"to_cuda","repo":"WHU-USI3DV/FreeReg","repo_kind":"official","path":"dataops/utils.py","file_url":"https://github.com/WHU-USI3DV/FreeReg/blob/HEAD/dataops/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ca32b40b0e54814"}},{"code_sha256_prefix":"e57cea746f89370d","entry":"compute_R_diff","repo":"WHU-USI3DV/FreeReg","repo_kind":"official","path":"utils/r_eval.py","file_url":"https://github.com/WHU-USI3DV/FreeReg/blob/HEAD/utils/r_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e57cea746f89370d"}},{"code_sha256_prefix":"2494bd08ee95716b","entry":"transform_points","repo":"WHU-USI3DV/FreeReg","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/WHU-USI3DV/FreeReg/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2494bd08ee95716b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}