{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-relation-alignment-for-calibrated","title":"Learning Relation Alignment for Calibrated Cross-modal Retrieval","arxiv_id":"2105.13868","date":"2021-05-28","proceeding":"ACL 2021 5","authors":["Shuhuai Ren","Junyang Lin","Guangxiang Zhao","Rui Men","An Yang","Jingren Zhou","Xu sun","Hongxia Yang"],"abstract":"Despite the achievements of large-scale multimodal pre-training approaches, cross-modal retrieval, e.g., image-text retrieval, remains a challenging task. To bridge the semantic gap between the two modalities, previous studies mainly focus on word-region alignment at the object level, lacking the matching between the linguistic relation among the words and the visual relation among the regions. The neglect of such relation consistency impairs the contextualized representation of image-text pairs and hinders the model performance and the interpretability. In this paper, we first propose a novel metric, Intra-modal Self-attention Distance (ISD), to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. In response, we present Inter-modal Alignment on Intra-modal Self-attentions (IAIS), a regularized training method to optimize the ISD and calibrate intra-modal self-attentions from the two modalities mutually via inter-modal alignment. The IAIS regularizer boosts the performance of prevailing models on Flickr30k and MS COCO datasets by a considerable margin, which demonstrates the superiority of our approach.","url_abs":"https://arxiv.org/abs/2105.13868v2","url_pdf":"https://arxiv.org/pdf/2105.13868v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-relation-alignment-for-calibrated","repo_url":"https://github.com/lancopku/IAIS","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"image-text-retrieval","task_name":"Image-text Retrieval"},{"task_slug":"image-to-text-retrieval","task_name":"Image-to-Text Retrieval"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-modal-retrieval-on-flickr30k","task":"Cross-Modal Retrieval","dataset":"Flickr30k","model":"IAIS","rank_in_archive_order":11,"of":27,"metrics":{"Image-to-text R@1":"88.3","Image-to-text R@10":"99.4","Image-to-text R@5":"98.4","Text-to-image R@1":"76.86","Text-to-image R@10":"95.72","Text-to-image R@5":"93.3"},"uses_additional_data":true},{"leaderboard":"/sota/image-to-text-retrieval-on-coco","task":"Image-to-Text Retrieval","dataset":"COCO (Common Objects in Context)","model":"IAIS","rank_in_archive_order":4,"of":9,"metrics":{"Recall@1":"67.78","Recall@10":"94.48","Recall@5":"89.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2105.13868","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.13868"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lancopku/IAIS","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/ChenRocks/UNITER","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fed42c63be2ceb6c","entry":"multi_head_attention_forward","repo":"lancopku/IAIS","repo_kind":"official","path":"model/attention.py","file_url":"https://github.com/lancopku/IAIS/blob/HEAD/model/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fed42c63be2ceb6c"}},{"code_sha256_prefix":"7a6b1633c8b59698","entry":"MultiheadAttention","repo":"lancopku/IAIS","repo_kind":"official","path":"model/attention.py","file_url":"https://github.com/lancopku/IAIS/blob/HEAD/model/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a6b1633c8b59698"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}