{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unlock-multi-modal-capability-of-dense","title":"MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin","arxiv_id":"2310.14037","date":"2023-10-21","proceeding":null,"authors":["Tianshuo Zhou","Sen Mei","Xinze Li","Zhenghao Liu","Chenyan Xiong","Zhiyuan Liu","Yu Gu","Ge Yu"],"abstract":"This paper proposes Multi-modAl Retrieval model via Visual modulE pLugin (MARVEL), which learns an embedding space for queries and multi-modal documents to conduct retrieval. MARVEL encodes queries and multi-modal documents with a unified encoder model, which helps to alleviate the modality gap between images and texts. Specifically, we enable the image understanding ability of the well-trained dense retriever, T5-ANCE, by incorporating the visual module's encoded image features as its inputs. To facilitate the multi-modal retrieval tasks, we build the ClueWeb22-MM dataset based on the ClueWeb22 dataset, which regards anchor texts as queries, and extracts the related text and image documents from anchor-linked web pages. Our experiments show that MARVEL significantly outperforms the state-of-the-art methods on the multi-modal retrieval dataset WebQA and ClueWeb22-MM. MARVEL provides an opportunity to broaden the advantages of text retrieval to the multi-modal scenario. Besides, we also illustrate that the language model has the ability to extract image semantics and partly map the image features to the input word embedding space. All codes are available at https://github.com/OpenMatch/MARVEL.","url_abs":"https://arxiv.org/abs/2310.14037v5","url_pdf":"https://arxiv.org/pdf/2310.14037v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unlock-multi-modal-capability-of-dense","repo_url":"https://github.com/openmatch/marvel","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.14037","atlas_url":"https://app.syntology.ai/?focus=2310.14037","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.14037"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openmatch/marvel","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":18},"by_repo_kind":{"official":{"samples":18,"ran":18,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5f50bc569621e9e0","entry":"convert_to_string_id","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/retrieval.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f50bc569621e9e0"}},{"code_sha256_prefix":"78797c5fbc1d7a65","entry":"get_img_patch_token_size","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/utils.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78797c5fbc1d7a65"}},{"code_sha256_prefix":"44dabd5275dfdd74","entry":"load_candidate_from_stream","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/msmarco_eval.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/msmarco_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"44dabd5275dfdd74"}},{"code_sha256_prefix":"54e02b2b07d569cc","entry":"load_caps","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/data.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54e02b2b07d569cc"}},{"code_sha256_prefix":"f3fa7721d1d6c94a","entry":"load_caps","repo":"openmatch/marvel","repo_kind":"official","path":"DPR/data.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/DPR/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3fa7721d1d6c94a"}},{"code_sha256_prefix":"1567be8b2bcb92d0","entry":"load_docs","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/data.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1567be8b2bcb92d0"}},{"code_sha256_prefix":"6b65342a5278b830","entry":"load_docs","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/gen_embeddings.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/gen_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b65342a5278b830"}},{"code_sha256_prefix":"81c9bc5cbd1e63f4","entry":"load_docs","repo":"openmatch/marvel","repo_kind":"official","path":"DPR/data.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/DPR/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81c9bc5cbd1e63f4"}},{"code_sha256_prefix":"735781aa80ba078f","entry":"load_docs","repo":"openmatch/marvel","repo_kind":"official","path":"DPR/gen_embeddings.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/DPR/gen_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"735781aa80ba078f"}},{"code_sha256_prefix":"b334283bb59e66d0","entry":"load_file","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/data.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b334283bb59e66d0"}},{"code_sha256_prefix":"d71b15e3f71f7d11","entry":"load_file","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/retrieval.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d71b15e3f71f7d11"}},{"code_sha256_prefix":"bc7b881120312440","entry":"load_file","repo":"openmatch/marvel","repo_kind":"official","path":"DPR/data.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/DPR/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc7b881120312440"}},{"code_sha256_prefix":"70926fb8a521dab4","entry":"load_file","repo":"openmatch/marvel","repo_kind":"official","path":"DPR/get_hard_negs_all.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/DPR/get_hard_negs_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70926fb8a521dab4"}},{"code_sha256_prefix":"33777073f6fbe9ee","entry":"load_img_docs","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/gen_embeddings.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/gen_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33777073f6fbe9ee"}},{"code_sha256_prefix":"8d62ec4c9d2a8b34","entry":"load_queries","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/gen_embeddings.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/gen_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d62ec4c9d2a8b34"}},{"code_sha256_prefix":"c82904a23f9f6650","entry":"load_queries","repo":"openmatch/marvel","repo_kind":"official","path":"DPR/gen_embeddings.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/DPR/gen_embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c82904a23f9f6650"}},{"code_sha256_prefix":"73a28fd25464e18b","entry":"load_reference","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/msmarco_eval.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/msmarco_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73a28fd25464e18b"}},{"code_sha256_prefix":"3bb6407c68aabca4","entry":"load_reference_from_stream","repo":"openmatch/marvel","repo_kind":"official","path":"ANCE/msmarco_eval.py","file_url":"https://github.com/openmatch/marvel/blob/HEAD/ANCE/msmarco_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3bb6407c68aabca4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}