{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cross-modal-active-complementary-learning-1","title":"Cross-modal Active Complementary Learning with Self-refining Correspondence","arxiv_id":"2310.17468","date":"2023-10-26","proceeding":"NeurIPS 2023 11","authors":["Yang Qin","Yuan Sun","Dezhong Peng","Joey Tianyi Zhou","Xi Peng","Peng Hu"],"abstract":"Recently, image-text matching has attracted more and more attention from academia and industry, which is fundamental to understanding the latent correspondence across visual and textual modalities. However, most existing methods implicitly assume the training pairs are well-aligned while ignoring the ubiquitous annotation noise, a.k.a noisy correspondence (NC), thereby inevitably leading to a performance drop. Although some methods attempt to address such noise, they still face two challenging problems: excessive memorizing/overfitting and unreliable correction for NC, especially under high noise. To address the two problems, we propose a generalized Cross-modal Robust Complementary Learning framework (CRCL), which benefits from a novel Active Complementary Loss (ACL) and an efficient Self-refining Correspondence Correction (SCC) to improve the robustness of existing methods. Specifically, ACL exploits active and complementary learning losses to reduce the risk of providing erroneous supervision, leading to theoretically and experimentally demonstrated robustness against NC. SCC utilizes multiple self-refining processes with momentum correction to enlarge the receptive field for correcting correspondences, thereby alleviating error accumulation and achieving accurate and stable corrections. We carry out extensive experiments on three image-text benchmarks, i.e., Flickr30K, MS-COCO, and CC152K, to verify the superior robustness of our CRCL against synthetic and real-world noisy correspondences.","url_abs":"https://arxiv.org/abs/2310.17468v2","url_pdf":"https://arxiv.org/pdf/2310.17468v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cross-modal-active-complementary-learning-1","repo_url":"https://github.com/qinyang79/crcl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"cross-modal-retrieval-with-noisy","task_name":"Cross-modal retrieval with noisy correspondence"},{"task_slug":"image-text-matching","task_name":"Image-text matching"},{"task_slug":"text-matching","task_name":"Text Matching"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-modal-retrieval-with-noisy-1","task":"Cross-modal retrieval with noisy correspondence","dataset":"CC152K","model":"CRCL","rank_in_archive_order":4,"of":15,"metrics":{"Image-to-text R@1":"41.8","Image-to-text R@10":"76.5","Image-to-text R@5":"67.4","R-Sum":"373.7","Text-to-image R@1":"41.6","Text-to-image R@10":"78.4","Text-to-image R@5":"68.0"},"uses_additional_data":false},{"leaderboard":"/sota/cross-modal-retrieval-with-noisy-3","task":"Cross-modal retrieval with noisy correspondence","dataset":"COCO-Noisy","model":"CRCL","rank_in_archive_order":5,"of":17,"metrics":{"Image-to-text R@1":"79.6","Image-to-text R@10":"98.7","Image-to-text R@5":"96.1","R-Sum":"525.6","Text-to-image R@1":"64.7","Text-to-image R@10":"95.9","Text-to-image R@5":"90.6"},"uses_additional_data":false},{"leaderboard":"/sota/cross-modal-retrieval-with-noisy-2","task":"Cross-modal retrieval with noisy correspondence","dataset":"Flickr30K-Noisy","model":"CRCL","rank_in_archive_order":4,"of":16,"metrics":{"Image-to-text R@1":"77.9","Image-to-text R@10":"98.3","Image-to-text R@5":"95.4","R-Sum":"507.8","Text-to-image R@1":"60.9","Text-to-image R@10":"90.6","Text-to-image R@5":"84.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.17468","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.17468"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/QinYang79/CRCL","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qinyang79/crcl","reach":{"status":"ok"}}],"summary":{"ran":15,"ran_draft_wrong":6,"ran_honours":2,"unverified":5},"by_repo_kind":{"official":{"samples":27,"ran":22,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":28,"samples":[{"code_sha256_prefix":"693b17df7b8c6055","entry":"AttentionFiltration","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"693b17df7b8c6055"}},{"code_sha256_prefix":"0822e67441847823","entry":"ContrastiveLoss","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0822e67441847823"}},{"code_sha256_prefix":"4075e94c929dd49b","entry":"EncoderImage","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4075e94c929dd49b"}},{"code_sha256_prefix":"23bb367e48dddb94","entry":"EncoderImageAggr","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23bb367e48dddb94"}},{"code_sha256_prefix":"87256eef10e84cf7","entry":"EncoderText","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87256eef10e84cf7"}},{"code_sha256_prefix":"e8fda28aefdcfcdb","entry":"GPO","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8fda28aefdcfcdb"}},{"code_sha256_prefix":"10426d360aea3d8e","entry":"MLP","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10426d360aea3d8e"}},{"code_sha256_prefix":"74500030ae6c1b43","entry":"SCAN_attention","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74500030ae6c1b43"}},{"code_sha256_prefix":"83d9349f26c10e82","entry":"TextSA","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"83d9349f26c10e82"}},{"code_sha256_prefix":"c09ebb6e3e75d64c","entry":"VisualSA","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c09ebb6e3e75d64c"}},{"code_sha256_prefix":"498e6d8b90b78468","entry":"collate_fn","repo":"QinYang79/CRCL","repo_kind":"official","path":"CRCL_NeurIPS23/data.py","file_url":"https://github.com/QinYang79/CRCL/blob/HEAD/CRCL_NeurIPS23/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"498e6d8b90b78468"}},{"code_sha256_prefix":"3a3ba65d84f3eeff","entry":"cosine_sim","repo":"QinYang79/CRCL","repo_kind":"official","path":"CRCL_NeurIPS23/model/SGRAF.py","file_url":"https://github.com/QinYang79/CRCL/blob/HEAD/CRCL_NeurIPS23/model/SGRAF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3a3ba65d84f3eeff"}},{"code_sha256_prefix":"7b18bd64b585e4d2","entry":"cosine_similarity_matrix","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b18bd64b585e4d2"}},{"code_sha256_prefix":"bdf65a1c26e5fec5","entry":"encode_data","repo":"QinYang79/CRCL","repo_kind":"official","path":"CRCL_NeurIPS23/evaluation.py","file_url":"https://github.com/QinYang79/CRCL/blob/HEAD/CRCL_NeurIPS23/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bdf65a1c26e5fec5"}},{"code_sha256_prefix":"414f20e131140e01","entry":"from_txt","repo":"QinYang79/CRCL","repo_kind":"official","path":"CRCL_NeurIPS23/vocab.py","file_url":"https://github.com/QinYang79/CRCL/blob/HEAD/CRCL_NeurIPS23/vocab.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"414f20e131140e01"}},{"code_sha256_prefix":"091fa44320344aae","entry":"get_precomp_loader","repo":"QinYang79/CRCL","repo_kind":"official","path":"CRCL_NeurIPS23/data.py","file_url":"https://github.com/QinYang79/CRCL/blob/HEAD/CRCL_NeurIPS23/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"091fa44320344aae"}},{"code_sha256_prefix":"42cf9ff13b24ef29","entry":"get_sim","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"42cf9ff13b24ef29"}},{"code_sha256_prefix":"b943467457f9825d","entry":"l1norm","repo":"QinYang79/CRCL","repo_kind":"official","path":"CRCL_NeurIPS23/model/SGRAF.py","file_url":"https://github.com/QinYang79/CRCL/blob/HEAD/CRCL_NeurIPS23/model/SGRAF.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b943467457f9825d"}},{"code_sha256_prefix":"9bf455ab09372ab4","entry":"l1norm","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9bf455ab09372ab4"}},{"code_sha256_prefix":"be7c4a676b9f1e19","entry":"l2norm","repo":"QinYang79/CRCL","repo_kind":"official","path":"CRCL_NeurIPS23/model/VSEinfty.py","file_url":"https://github.com/QinYang79/CRCL/blob/HEAD/CRCL_NeurIPS23/model/VSEinfty.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be7c4a676b9f1e19"}},{"code_sha256_prefix":"03e1f23d59a7c45a","entry":"l2norm","repo":"QinYang79/CRCL","repo_kind":"official","path":"CRCL_NeurIPS23/model/SGRAF.py","file_url":"https://github.com/QinYang79/CRCL/blob/HEAD/CRCL_NeurIPS23/model/SGRAF.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03e1f23d59a7c45a"}},{"code_sha256_prefix":"66c401f1c3fade5d","entry":"positional_encoding_1d","repo":"QinYang79/CRCL","repo_kind":"official","path":"CRCL_NeurIPS23/model/VSEinfty.py","file_url":"https://github.com/QinYang79/CRCL/blob/HEAD/CRCL_NeurIPS23/model/VSEinfty.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66c401f1c3fade5d"}},{"code_sha256_prefix":"7a82222075ccfaac","entry":"shard_attn_scores","repo":"QinYang79/CRCL","repo_kind":"official","path":"CRCL_NeurIPS23/evaluation.py","file_url":"https://github.com/QinYang79/CRCL/blob/HEAD/CRCL_NeurIPS23/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a82222075ccfaac"}},{"code_sha256_prefix":"52df4f2c82114f8a","entry":"CRCL","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"52df4f2c82114f8a"}},{"code_sha256_prefix":"1cd408d8277e77a1","entry":"EncoderSimilarity","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1cd408d8277e77a1"}},{"code_sha256_prefix":"d99c8e38277ee8a9","entry":"SGRAF","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d99c8e38277ee8a9"}},{"code_sha256_prefix":"615b05a63a1c5cde","entry":"VSEModel","repo":"qinyang79/crcl","repo_kind":"official","path":"CRCL_NeurIPS23/model/CRCL.py","file_url":"https://github.com/qinyang79/crcl/blob/HEAD/CRCL_NeurIPS23/model/CRCL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"615b05a63a1c5cde"}},{"code_sha256_prefix":"c2c0455d61babb92","entry":"get_loaders","repo":"QinYang79/CRCL","repo_kind":"official","path":"CRCL_NeurIPS23/data.py","file_url":"https://github.com/QinYang79/CRCL/blob/HEAD/CRCL_NeurIPS23/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2c0455d61babb92"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}