{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stacked-cross-attention-for-image-text","title":"Stacked Cross Attention for Image-Text Matching","arxiv_id":"1803.08024","date":"2018-03-21","proceeding":"ECCV 2018 9","authors":["Kuang-Huei Lee","Xi Chen","Gang Hua","Houdong Hu","Xiaodong He"],"abstract":"In this paper, we study the problem of image-text matching. Inferring the\nlatent semantic alignment between objects or other salient stuff (e.g. snow,\nsky, lawn) and the corresponding words in sentences allows to capture\nfine-grained interplay between vision and language, and makes image-text\nmatching more interpretable. Prior work either simply aggregates the similarity\nof all possible pairs of regions and words without attending differentially to\nmore and less important words or regions, or uses a multi-step attentional\nprocess to capture limited number of semantic alignments which is less\ninterpretable. In this paper, we present Stacked Cross Attention to discover\nthe full latent alignments using both image regions and words in a sentence as\ncontext and infer image-text similarity. Our approach achieves the\nstate-of-the-art results on the MS-COCO and Flickr30K datasets. On Flickr30K,\nour approach outperforms the current best methods by 22.1% relatively in text\nretrieval from image query, and 18.2% relatively in image retrieval with text\nquery (based on Recall@1). On MS-COCO, our approach improves sentence retrieval\nby 17.8% relatively and image retrieval by 16.6% relatively (based on Recall@1\nusing the 5K test set). Code has been made available at:\nhttps://github.com/kuanghuei/SCAN.","url_abs":"http://arxiv.org/abs/1803.08024v2","url_pdf":"http://arxiv.org/pdf/1803.08024v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stacked-cross-attention-for-image-text","repo_url":"https://github.com/kuanghuei/SCAN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"stacked-cross-attention-for-image-text","repo_url":"https://github.com/MysteryVaibhav/SCAN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"stacked-cross-attention-for-image-text","repo_url":"https://github.com/abhidipbhattacharyya/srl_aware_ret","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"stacked-cross-attention-for-image-text","repo_url":"https://github.com/adlnlp/attention_vl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"stacked-cross-attention-for-image-text","repo_url":"https://github.com/idejie/SCAN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"stacked-cross-attention-for-image-text","repo_url":"https://github.com/hthoai/image-text-matching","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"image-text-matching","task_name":"Image-text matching"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-retrieval","task_name":"Sentence Retrieval"},{"task_slug":"text-matching","task_name":"Text Matching"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"text-similarity","task_name":"text similarity"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-modal-retrieval-on-coco-2014","task":"Cross-Modal Retrieval","dataset":"COCO 2014","model":"SCAN","rank_in_archive_order":32,"of":36,"metrics":{"Image-to-text R@1":"50.4","Image-to-text R@10":"90.0","Image-to-text R@5":"82.2","Text-to-image R@1":"38.6","Text-to-image R@10":"80.4","Text-to-image R@5":"69.3"},"uses_additional_data":false},{"leaderboard":"/sota/cross-modal-retrieval-on-flickr30k","task":"Cross-Modal Retrieval","dataset":"Flickr30k","model":"SCAN","rank_in_archive_order":21,"of":27,"metrics":{"Image-to-text R@1":"67.4","Image-to-text R@10":"95.8","Image-to-text R@5":"90.3","Text-to-image R@1":"48.6","Text-to-image R@10":"85.2","Text-to-image R@5":"77.7"},"uses_additional_data":false},{"leaderboard":"/sota/image-retrieval-on-flickr30k-1k-test","task":"Image Retrieval","dataset":"Flickr30K 1K test","model":"SCAN i-t","rank_in_archive_order":10,"of":18,"metrics":{"R@1":"44.0","R@10":"82.6","R@5":"74.2"},"uses_additional_data":false},{"leaderboard":"/sota/image-retrieval-on-photochat","task":"Image Retrieval","dataset":"PhotoChat","model":"SCAN","rank_in_archive_order":4,"of":5,"metrics":{"R1":"10.4","R@10":"37.1","R@5":"27.0","Sum(R@1,5,10)":"74.5"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1803.08024","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1803.08024"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abhidipbhattacharyya/srl_aware_ret","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MysteryVaibhav/SCAN","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adlnlp/attention_vl","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hthoai/image-text-matching","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kuanghuei/SCAN","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/idejie/SCAN","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":5,"ran_fixture":1,"ran_honours":1,"unverified":9},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":15,"ran":7,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"cdee6932aa05022e","entry":"deserialize_vocab","repo":"abhidipbhattacharyya/srl_aware_ret","repo_kind":"listed","path":"vocab.py","file_url":"https://github.com/abhidipbhattacharyya/srl_aware_ret/blob/HEAD/vocab.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdee6932aa05022e"}},{"code_sha256_prefix":"1119155cc37cafa8","entry":"encode_data","repo":"abhidipbhattacharyya/srl_aware_ret","repo_kind":"listed","path":"evaluation.py","file_url":"https://github.com/abhidipbhattacharyya/srl_aware_ret/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1119155cc37cafa8"}},{"code_sha256_prefix":"ef05dda21c1cc726","entry":"from_txt","repo":"abhidipbhattacharyya/srl_aware_ret","repo_kind":"listed","path":"vocab.py","file_url":"https://github.com/abhidipbhattacharyya/srl_aware_ret/blob/HEAD/vocab.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef05dda21c1cc726"}},{"code_sha256_prefix":"0da42e0e379224b0","entry":"from_txt_vocab","repo":"abhidipbhattacharyya/srl_aware_ret","repo_kind":"listed","path":"vocab.py","file_url":"https://github.com/abhidipbhattacharyya/srl_aware_ret/blob/HEAD/vocab.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0da42e0e379224b0"}},{"code_sha256_prefix":"05aa9e5fd7ec0e01","entry":"process_sim","repo":"abhidipbhattacharyya/srl_aware_ret","repo_kind":"listed","path":"evaluation.py","file_url":"https://github.com/abhidipbhattacharyya/srl_aware_ret/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"05aa9e5fd7ec0e01"}},{"code_sha256_prefix":"68f208c3005da07a","entry":"read_all_images","repo":"hthoai/image-text-matching","repo_kind":"listed","path":"utils/read_all_images.py","file_url":"https://github.com/hthoai/image-text-matching/blob/HEAD/utils/read_all_images.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68f208c3005da07a"}},{"code_sha256_prefix":"652f07c5ff410713","entry":"softmax","repo":"abhidipbhattacharyya/srl_aware_ret","repo_kind":"listed","path":"evaluation.py","file_url":"https://github.com/abhidipbhattacharyya/srl_aware_ret/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"652f07c5ff410713"}},{"code_sha256_prefix":"07c90e643974e90d","entry":"EncoderImage","repo":"idejie/SCAN","repo_kind":"listed","path":"model.py","file_url":"https://github.com/idejie/SCAN/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"07c90e643974e90d"}},{"code_sha256_prefix":"7b9b11b5023a6ca4","entry":"build_vocab","repo":"idejie/SCAN","repo_kind":"listed","path":"vocab.py","file_url":"https://github.com/idejie/SCAN/blob/HEAD/vocab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b9b11b5023a6ca4"}},{"code_sha256_prefix":"eb35209092068c75","entry":"collate_fn","repo":"idejie/SCAN","repo_kind":"listed","path":"data.py","file_url":"https://github.com/idejie/SCAN/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eb35209092068c75"}},{"code_sha256_prefix":"77cfc8299b5facec","entry":"encode_data","repo":"kuanghuei/SCAN","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/kuanghuei/SCAN/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77cfc8299b5facec"}},{"code_sha256_prefix":"647842a3913aa59c","entry":"encode_data","repo":"idejie/SCAN","repo_kind":"listed","path":"evaluation.py","file_url":"https://github.com/idejie/SCAN/blob/HEAD/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"647842a3913aa59c"}},{"code_sha256_prefix":"a3b22af17bee87cc","entry":"get_loaders","repo":"idejie/SCAN","repo_kind":"listed","path":"data.py","file_url":"https://github.com/idejie/SCAN/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a3b22af17bee87cc"}},{"code_sha256_prefix":"f8b728dce3e79700","entry":"get_precomp_loader","repo":"idejie/SCAN","repo_kind":"listed","path":"data.py","file_url":"https://github.com/idejie/SCAN/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8b728dce3e79700"}},{"code_sha256_prefix":"a1e099b0f700be6a","entry":"l1norm","repo":"idejie/SCAN","repo_kind":"listed","path":"model.py","file_url":"https://github.com/idejie/SCAN/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1e099b0f700be6a"}},{"code_sha256_prefix":"fe7440ba7662dcfc","entry":"l2norm","repo":"idejie/SCAN","repo_kind":"listed","path":"model.py","file_url":"https://github.com/idejie/SCAN/blob/HEAD/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe7440ba7662dcfc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}