{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/generative-cross-modal-retrieval-memorizing","title":"Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond","arxiv_id":"2402.10805","date":"2024-02-16","proceeding":null,"authors":["Yongqi Li","Wenjie Wang","Leigang Qu","Liqiang Nie","Wenjie Li","Tat-Seng Chua"],"abstract":"The recent advancements in generative language models have demonstrated their ability to memorize knowledge from documents and recall knowledge to respond to user queries effectively. Building upon this capability, we propose to enable multimodal large language models (MLLMs) to memorize and recall images within their parameters. Given a user query for visual content, the MLLM is anticipated to \"recall\" the relevant image from its parameters as the response. Achieving this target presents notable challenges, including inbuilt visual memory and visual recall schemes within MLLMs. To address these challenges, we introduce a generative cross-modal retrieval framework, which assigns unique identifier strings to represent images and involves two training steps: learning to memorize and learning to retrieve. The first step focuses on training the MLLM to memorize the association between images and their respective identifiers. The latter step teaches the MLLM to generate the corresponding identifier of the target image, given the textual query input. By memorizing images in MLLMs, we introduce a new paradigm to cross-modal retrieval, distinct from previous discriminative approaches. The experiments demonstrate that the generative paradigm performs effectively and efficiently even with large-scale image candidate sets.","url_abs":"https://arxiv.org/abs/2402.10805v1","url_pdf":"https://arxiv.org/pdf/2402.10805v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.10805","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.10805"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/liyongqi67/GRACE","reach":null}],"summary":{"ran":3,"unverified":9},"by_repo_kind":{"found_in_text":{"samples":12,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"81667a1b9711fecf","entry":"GatedCrossAttentionBlock","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81667a1b9711fecf"}},{"code_sha256_prefix":"a517fe2bc0cafd0f","entry":"MaskedCrossAttention","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a517fe2bc0cafd0f"}},{"code_sha256_prefix":"bda9925505979900","entry":"PerceiverAttention","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bda9925505979900"}},{"code_sha256_prefix":"2440d35032fda754","entry":"CrossAttentionMixin","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2440d35032fda754"}},{"code_sha256_prefix":"e2a64ac7814b0a7e","entry":"DecoderLayerWithCrossAttention","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2a64ac7814b0a7e"}},{"code_sha256_prefix":"a4166fca23b64924","entry":"Flamingo","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4166fca23b64924"}},{"code_sha256_prefix":"c7929ab2509b5011","entry":"PerceiverResampler","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7929ab2509b5011"}},{"code_sha256_prefix":"32419e68bc383cf5","entry":"VLM","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32419e68bc383cf5"}},{"code_sha256_prefix":"e15a136b2df3c5fa","entry":"VLMOutputWithPast","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e15a136b2df3c5fa"}},{"code_sha256_prefix":"0ccbc1d02b3ade24","entry":"VLMWithCrossAttention","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ccbc1d02b3ade24"}},{"code_sha256_prefix":"96919fc2f4245c57","entry":"extend_instance","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96919fc2f4245c57"}},{"code_sha256_prefix":"d84729d3d1bb1d83","entry":"setattr_recursive","repo":"liyongqi67/GRACE","repo_kind":"found_in_text","path":"open_flamingo_deepspeed/open_flamingo/src/flamingo.py","file_url":"https://github.com/liyongqi67/GRACE/blob/HEAD/open_flamingo_deepspeed/open_flamingo/src/flamingo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d84729d3d1bb1d83"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}