{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transformer-reasoning-network-for-image-text","title":"Transformer Reasoning Network for Image-Text Matching and Retrieval","arxiv_id":"2004.09144","date":"2020-04-20","proceeding":null,"authors":["Nicola Messina","Fabrizio Falchi","Andrea Esuli","Giuseppe Amato"],"abstract":"Image-text matching is an interesting and fascinating task in modern AI research. Despite the evolution of deep-learning-based image and text processing systems, multi-modal matching remains a challenging problem. In this work, we consider the problem of accurate image-text matching for the task of multi-modal large-scale information retrieval. State-of-the-art results in image-text matching are achieved by inter-playing image and text features from the two different processing pipelines, usually using mutual attention mechanisms. However, this invalidates any chance to extract separate visual and textual features needed for later indexing steps in large-scale retrieval systems. In this regard, we introduce the Transformer Encoder Reasoning Network (TERN), an architecture built upon one of the modern relationship-aware self-attentive architectures, the Transformer Encoder (TE). This architecture is able to separately reason on the two different modalities and to enforce a final common abstract concept space by sharing the weights of the deeper transformer layers. Thanks to this design, the implemented network is able to produce compact and very rich visual and textual features available for the successive indexing step. Experiments are conducted on the MS-COCO dataset, and we evaluate the results using a discounted cumulative gain metric with relevance computed exploiting caption similarities, in order to assess possibly non-exact but relevant search results. We demonstrate that on this metric we are able to achieve state-of-the-art results in the image retrieval task. Our code is freely available at https://github.com/mesnico/TERN.","url_abs":"https://arxiv.org/abs/2004.09144v3","url_pdf":"https://arxiv.org/pdf/2004.09144v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transformer-reasoning-network-for-image-text","repo_url":"https://github.com/mesnico/TERN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"image-text-matching","task_name":"Image-text matching"},{"task_slug":"information-retrieval","task_name":"Information Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-matching","task_name":"Text Matching"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.09144","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.09144"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mesnico/TERN","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":1,"ran_draft_wrong":1,"unverified":12},"by_repo_kind":{"official":{"samples":14,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"894b253f35765276","entry":"generate_square_subsequent_mask","repo":"mesnico/TERN","repo_kind":"official","path":"models/utils.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/models/utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"894b253f35765276"}},{"code_sha256_prefix":"f92e514b398e75d8","entry":"l2norm","repo":"mesnico/TERN","repo_kind":"official","path":"models/utils.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/models/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f92e514b398e75d8"}},{"code_sha256_prefix":"ef1f72eefcd0fcda","entry":"EncoderText","repo":"mesnico/TERN","repo_kind":"official","path":"models/text.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/models/text.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef1f72eefcd0fcda"}},{"code_sha256_prefix":"c9ccc8160ca8229a","entry":"average_precision_score","repo":"mesnico/TERN","repo_kind":"official","path":"evaluate_utils/dcg.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/evaluate_utils/dcg.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9ccc8160ca8229a"}},{"code_sha256_prefix":"f6b4594fd6677dbe","entry":"dcg_score","repo":"mesnico/TERN","repo_kind":"official","path":"evaluate_utils/dcg.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/evaluate_utils/dcg.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6b4594fd6677dbe"}},{"code_sha256_prefix":"6a49bab714ec6122","entry":"dot_sim","repo":"mesnico/TERN","repo_kind":"official","path":"models/loss.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/models/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6a49bab714ec6122"}},{"code_sha256_prefix":"0919fda55c323689","entry":"encode_data","repo":"mesnico/TERN","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0919fda55c323689"}},{"code_sha256_prefix":"6e963f0bd8b4ec0f","entry":"find_nhead","repo":"mesnico/TERN","repo_kind":"official","path":"models/visual.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/models/visual.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e963f0bd8b4ec0f"}},{"code_sha256_prefix":"ae1d8e2720a00989","entry":"get_paths","repo":"mesnico/TERN","repo_kind":"official","path":"data.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ae1d8e2720a00989"}},{"code_sha256_prefix":"e7518699cbc1bd68","entry":"get_transform","repo":"mesnico/TERN","repo_kind":"official","path":"data.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7518699cbc1bd68"}},{"code_sha256_prefix":"d9047caaedf4aadb","entry":"my_collate","repo":"mesnico/TERN","repo_kind":"official","path":"evaluate_utils/compute_relevance.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/evaluate_utils/compute_relevance.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9047caaedf4aadb"}},{"code_sha256_prefix":"8e177567218199c4","entry":"my_lcs","repo":"mesnico/TERN","repo_kind":"official","path":"evaluate_utils/rouge.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/evaluate_utils/rouge.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8e177567218199c4"}},{"code_sha256_prefix":"da16023b8239c604","entry":"order_sim","repo":"mesnico/TERN","repo_kind":"official","path":"models/loss.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/models/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da16023b8239c604"}},{"code_sha256_prefix":"8183b31354ef5eef","entry":"ranking_precision_score","repo":"mesnico/TERN","repo_kind":"official","path":"evaluate_utils/dcg.py","file_url":"https://github.com/mesnico/TERN/blob/HEAD/evaluate_utils/dcg.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8183b31354ef5eef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}