{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-rematch-mismatched-pairs-for","title":"Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval","arxiv_id":"2403.05105","date":"2024-03-08","proceeding":"CVPR 2024 1","authors":["Haochen Han","Qinghua Zheng","Guang Dai","Minnan Luo","Jingdong Wang"],"abstract":"Collecting well-matched multimedia datasets is crucial for training cross-modal retrieval models. However, in real-world scenarios, massive multimodal data are harvested from the Internet, which inevitably contains Partially Mismatched Pairs (PMPs). Undoubtedly, such semantical irrelevant data will remarkably harm the cross-modal retrieval performance. Previous efforts tend to mitigate this problem by estimating a soft correspondence to down-weight the contribution of PMPs. In this paper, we aim to address this challenge from a new perspective: the potential semantic similarity among unpaired samples makes it possible to excavate useful knowledge from mismatched pairs. To achieve this, we propose L2RM, a general framework based on Optimal Transport (OT) that learns to rematch mismatched pairs. In detail, L2RM aims to generate refined alignments by seeking a minimal-cost transport plan across different modalities. To formalize the rematching idea in OT, first, we propose a self-supervised cost function that automatically learns from explicit similarity-cost mapping relation. Second, we present to model a partial OT problem while restricting the transport among false positives to further boost refined alignments. Extensive experiments on three benchmarks demonstrate our L2RM significantly improves the robustness against PMPs for existing models. The code is available at https://github.com/hhc1997/L2RM.","url_abs":"https://arxiv.org/abs/2403.05105v1","url_pdf":"https://arxiv.org/pdf/2403.05105v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-rematch-mismatched-pairs-for","repo_url":"https://github.com/hhc1997/l2rm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"cross-modal-retrieval-with-noisy","task_name":"Cross-modal retrieval with noisy correspondence"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-modal-retrieval-with-noisy-1","task":"Cross-modal retrieval with noisy correspondence","dataset":"CC152K","model":"L2RM-SGRAF","rank_in_archive_order":3,"of":15,"metrics":{"Image-to-text R@1":"43.0","Image-to-text R@10":"75.7","Image-to-text R@5":"67.5","R-Sum":"374.2","Text-to-image R@1":"42.8","Text-to-image R@10":"77.2","Text-to-image R@5":"68.0"},"uses_additional_data":false},{"leaderboard":"/sota/cross-modal-retrieval-with-noisy-3","task":"Cross-modal retrieval with noisy correspondence","dataset":"COCO-Noisy","model":"L2RM-SCARF","rank_in_archive_order":7,"of":17,"metrics":{"Image-to-text R@1":"80.2","Image-to-text R@10":"98.5","Image-to-text R@5":"96.3","R-Sum":"524.7","Text-to-image R@1":"64.2","Text-to-image R@10":"95.4","Text-to-image R@5":"90.1"},"uses_additional_data":false},{"leaderboard":"/sota/cross-modal-retrieval-with-noisy-2","task":"Cross-modal retrieval with noisy correspondence","dataset":"Flickr30K-Noisy","model":"L2RM-SGRAF","rank_in_archive_order":10,"of":16,"metrics":{"Image-to-text R@1":"77.9","Image-to-text R@10":"97.8","Image-to-text R@5":"95.2","R-Sum":"503.8","Text-to-image R@1":"59.8","Text-to-image R@10":"89.5","Text-to-image R@5":"83.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.05105","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.05105"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hhc1997/l2rm","reach":null}],"summary":{"ran":8,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"432d59115f341570","entry":"ContrastiveLoss","repo":"hhc1997/l2rm","repo_kind":"official","path":"models.py","file_url":"https://github.com/hhc1997/l2rm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"432d59115f341570"}},{"code_sha256_prefix":"28cdced0a58c81a0","entry":"GraphReasoning","repo":"hhc1997/l2rm","repo_kind":"official","path":"models.py","file_url":"https://github.com/hhc1997/l2rm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28cdced0a58c81a0"}},{"code_sha256_prefix":"ab912266807ca3b6","entry":"InfoNCE","repo":"hhc1997/l2rm","repo_kind":"official","path":"models.py","file_url":"https://github.com/hhc1997/l2rm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab912266807ca3b6"}},{"code_sha256_prefix":"17c042b57a89f173","entry":"JS_Divergence","repo":"hhc1997/l2rm","repo_kind":"official","path":"models.py","file_url":"https://github.com/hhc1997/l2rm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17c042b57a89f173"}},{"code_sha256_prefix":"de6018f9861ffa0e","entry":"KL_Divergence","repo":"hhc1997/l2rm","repo_kind":"official","path":"models.py","file_url":"https://github.com/hhc1997/l2rm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de6018f9861ffa0e"}},{"code_sha256_prefix":"3179f46a92ef7f79","entry":"RCE","repo":"hhc1997/l2rm","repo_kind":"official","path":"models.py","file_url":"https://github.com/hhc1997/l2rm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3179f46a92ef7f79"}},{"code_sha256_prefix":"6ba90668a9ac0fd7","entry":"Soft_InfoNCE","repo":"hhc1997/l2rm","repo_kind":"official","path":"models.py","file_url":"https://github.com/hhc1997/l2rm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ba90668a9ac0fd7"}},{"code_sha256_prefix":"0d8402b5f472c72f","entry":"sym_KL_Divergence","repo":"hhc1997/l2rm","repo_kind":"official","path":"models.py","file_url":"https://github.com/hhc1997/l2rm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d8402b5f472c72f"}},{"code_sha256_prefix":"2acfe96d5e542050","entry":"EncoderSimilarity","repo":"hhc1997/l2rm","repo_kind":"official","path":"models.py","file_url":"https://github.com/hhc1997/l2rm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2acfe96d5e542050"}},{"code_sha256_prefix":"436eaeec96ae09dd","entry":"SGRAF","repo":"hhc1997/l2rm","repo_kind":"official","path":"models.py","file_url":"https://github.com/hhc1997/l2rm/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"436eaeec96ae09dd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}