{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gim-learning-generalizable-image-matcher-from","title":"GIM: Learning Generalizable Image Matcher From Internet Videos","arxiv_id":"2402.11095","date":"2024-02-16","proceeding":null,"authors":["Xuelun Shen","Zhipeng Cai","Wei Yin","Matthias Müller","Zijun Li","Kaixuan Wang","Xiaozhi Chen","Cheng Wang"],"abstract":"Image matching is a fundamental computer vision problem. While learning-based methods achieve state-of-the-art performance on existing benchmarks, they generalize poorly to in-the-wild images. Such methods typically need to train separate models for different scene types and are impractical when the scene type is unknown in advance. One of the underlying problems is the limited scalability of existing data construction pipelines, which limits the diversity of standard image matching datasets. To address this problem, we propose GIM, a self-training framework for learning a single generalizable model based on any image matching architecture using internet videos, an abundant and diverse data source. Given an architecture, GIM first trains it on standard domain-specific datasets and then combines it with complementary matching methods to create dense labels on nearby frames of novel videos. These labels are filtered by robust fitting, and then enhanced by propagating them to distant frames. The final model is trained on propagated data with strong augmentations. We also propose ZEB, the first zero-shot evaluation benchmark for image matching. By mixing data from diverse domains, ZEB can thoroughly assess the cross-domain generalization performance of different methods. Applying GIM consistently improves the zero-shot performance of 3 state-of-the-art image matching architectures; with 50 hours of YouTube videos, the relative zero-shot performance improves by 8.4%-18.1%. GIM also enables generalization to extreme cross-domain data such as Bird Eye View (BEV) images of projected 3D point clouds (Fig. 1(c)). More importantly, our single zero-shot model consistently outperforms domain-specific baselines when evaluated on downstream tasks inherent to their respective domains. The video presentation is available at https://www.youtube.com/watch?v=FU_MJLD8LeY.","url_abs":"https://arxiv.org/abs/2402.11095v1","url_pdf":"https://arxiv.org/pdf/2402.11095v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gim-learning-generalizable-image-matcher-from","repo_url":"https://github.com/xuelunshen/gim","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-reconstruction","task_name":"3D Reconstruction"},{"task_slug":"camera-pose-estimation","task_name":"Camera Pose Estimation"},{"task_slug":"image-matching","task_name":"Image Matching"},{"task_slug":"keypoint-detection-and-image-matching","task_name":"Keypoint detection and image matching"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"visual-localization","task_name":"Visual Localization"}],"methods":[],"datasets_introduced":[{"slug":"zeb","name":"ZEB","full_name":"Zero-shot Evaluation Benchmark"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-matching-on-zeb","task":"Image Matching","dataset":"ZEB","model":"GIM-RoMa","rank_in_archive_order":1,"of":10,"metrics":{"Mean AUC@5°":"53.3"},"uses_additional_data":false},{"leaderboard":"/sota/image-matching-on-zeb","task":"Image Matching","dataset":"ZEB","model":"GIM-DKM","rank_in_archive_order":2,"of":10,"metrics":{"Mean AUC@5°":"51.2"},"uses_additional_data":false},{"leaderboard":"/sota/image-matching-on-zeb","task":"Image Matching","dataset":"ZEB","model":"GIM-LoFTR","rank_in_archive_order":5,"of":10,"metrics":{"Mean AUC@5°":"39.1"},"uses_additional_data":false},{"leaderboard":"/sota/image-matching-on-zeb","task":"Image Matching","dataset":"ZEB","model":"GIM-LightGlue","rank_in_archive_order":6,"of":10,"metrics":{"Mean AUC@5°":"38.3"},"uses_additional_data":false},{"leaderboard":"/sota/pose-estimation-on-inloc","task":"Pose Estimation","dataset":"InLoc","model":"GIM-DKM","rank_in_archive_order":1,"of":6,"metrics":{"DUC1-Acc@0.25m,10°":"57.1","DUC1-Acc@0.5m,10°":"78.8","DUC1-Acc@1.0m,10°":"88.4","DUC2-Acc@0.25m,10°":"70.2","DUC2-Acc@0.5m,10°":"91.6","DUC2-Acc@1.0m,10°":"92.4"},"uses_additional_data":false},{"leaderboard":"/sota/pose-estimation-on-inloc","task":"Pose Estimation","dataset":"InLoc","model":"GIM-LoFTR","rank_in_archive_order":2,"of":6,"metrics":{"DUC1-Acc@0.25m,10°":"54.5","DUC1-Acc@0.5m,10°":"78.3","DUC1-Acc@1.0m,10°":"87.4","DUC2-Acc@0.25m,10°":"63.4","DUC2-Acc@0.5m,10°":"83.2","DUC2-Acc@1.0m,10°":"87.0"},"uses_additional_data":false},{"leaderboard":"/sota/pose-estimation-on-inloc","task":"Pose Estimation","dataset":"InLoc","model":"GIM-SuperGlue","rank_in_archive_order":3,"of":6,"metrics":{"DUC1-Acc@0.25m,10°":"53.5","DUC1-Acc@0.5m,10°":"76.8","DUC1-Acc@1.0m,10°":"86.9","DUC2-Acc@0.25m,10°":"61.8","DUC2-Acc@0.5m,10°":"85.5","DUC2-Acc@1.0m,10°":"87.8"},"uses_additional_data":false},{"leaderboard":"/sota/visual-localization-on-aachen-day-night-v1-1","task":"Visual Localization","dataset":"Aachen Day-Night v1.1 Benchmark","model":"GIM-LoFTR","rank_in_archive_order":1,"of":7,"metrics":{"Acc@0.25m, 2°":"79.1","Acc@0.5m, 5°":"91.6","Acc@5m, 10°":"100.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-localization-on-aachen-day-night-v1-1","task":"Visual Localization","dataset":"Aachen Day-Night v1.1 Benchmark","model":"GIM-SuperGlue","rank_in_archive_order":3,"of":7,"metrics":{"Acc@0.25m, 2°":"78.0","Acc@0.5m, 5°":"90.6","Acc@5m, 10°":"100.0"},"uses_additional_data":false},{"leaderboard":"/sota/visual-localization-on-aachen-day-night-v1-1","task":"Visual Localization","dataset":"Aachen Day-Night v1.1 Benchmark","model":"GIM-DKM","rank_in_archive_order":5,"of":7,"metrics":{"Acc@0.25m, 2°":"77.0","Acc@0.5m, 5°":"90.1","Acc@5m, 10°":"99.5"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.11095","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.11095"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xuelunshen/gim","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":12,"unverified":5},"by_repo_kind":{"official":{"samples":17,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6b4df3b8d1f95653","entry":"assign_keypoints","repo":"xuelunshen/gim","repo_kind":"official","path":"hloc/match_dense.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/hloc/match_dense.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b4df3b8d1f95653"}},{"code_sha256_prefix":"a6b2a9c667b63a4c","entry":"build_augmentor","repo":"xuelunshen/gim","repo_kind":"official","path":"datasets/augment.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/datasets/augment.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6b2a9c667b63a4c"}},{"code_sha256_prefix":"afe48ee65c2a83aa","entry":"camera_intrinsic_transform","repo":"xuelunshen/gim","repo_kind":"official","path":"datasets/scenenet/utils.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/datasets/scenenet/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afe48ee65c2a83aa"}},{"code_sha256_prefix":"12c09a1a10ca585d","entry":"collate_fn","repo":"xuelunshen/gim","repo_kind":"official","path":"datasets/data.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/datasets/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"12c09a1a10ca585d"}},{"code_sha256_prefix":"3e3aed34677c7943","entry":"get_divisible_wh","repo":"xuelunshen/gim","repo_kind":"official","path":"video_preprocessor.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/video_preprocessor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e3aed34677c7943"}},{"code_sha256_prefix":"2bcc62914d8b4ce5","entry":"get_resized_wh","repo":"xuelunshen/gim","repo_kind":"official","path":"video_preprocessor.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/video_preprocessor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2bcc62914d8b4ce5"}},{"code_sha256_prefix":"21abf193b3d22564","entry":"get_resized_wh","repo":"xuelunshen/gim","repo_kind":"official","path":"datasets/utils.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/datasets/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"21abf193b3d22564"}},{"code_sha256_prefix":"c736348f2c26f6a4","entry":"imread_color","repo":"xuelunshen/gim","repo_kind":"official","path":"datasets/utils.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/datasets/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c736348f2c26f6a4"}},{"code_sha256_prefix":"28b32b7ce4eceafa","entry":"imread_gray","repo":"xuelunshen/gim","repo_kind":"official","path":"datasets/utils.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/datasets/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"28b32b7ce4eceafa"}},{"code_sha256_prefix":"ef0704c1e391fa96","entry":"position_to_np_array","repo":"xuelunshen/gim","repo_kind":"official","path":"datasets/scenenet/utils.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/datasets/scenenet/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef0704c1e391fa96"}},{"code_sha256_prefix":"f983f6b4bdc58b8d","entry":"read_deeplab_image","repo":"xuelunshen/gim","repo_kind":"official","path":"video_preprocessor.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/video_preprocessor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f983f6b4bdc58b8d"}},{"code_sha256_prefix":"34e02c0f95a179a4","entry":"read_depth","repo":"xuelunshen/gim","repo_kind":"official","path":"datasets/scenenet/utils.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/datasets/scenenet/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"34e02c0f95a179a4"}},{"code_sha256_prefix":"7580602299bba28c","entry":"error_auc","repo":"xuelunshen/gim","repo_kind":"official","path":"analysis.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7580602299bba28c"}},{"code_sha256_prefix":"903ef22b452ab670","entry":"get_grouped_ids","repo":"xuelunshen/gim","repo_kind":"official","path":"hloc/match_dense.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/hloc/match_dense.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"903ef22b452ab670"}},{"code_sha256_prefix":"5f12182269ed0b51","entry":"main","repo":"xuelunshen/gim","repo_kind":"official","path":"hloc/extract_features.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/hloc/extract_features.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f12182269ed0b51"}},{"code_sha256_prefix":"47ffaeadde70c45c","entry":"resize_image","repo":"xuelunshen/gim","repo_kind":"official","path":"hloc/extract_features.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/hloc/extract_features.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47ffaeadde70c45c"}},{"code_sha256_prefix":"e54750836225cbe0","entry":"to_cpts","repo":"xuelunshen/gim","repo_kind":"official","path":"hloc/match_dense.py","file_url":"https://github.com/xuelunshen/gim/blob/HEAD/hloc/match_dense.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e54750836225cbe0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}