{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/look-twice-before-you-answer-memory-space","title":"Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models","arxiv_id":"2410.03577","date":"2024-10-04","proceeding":null,"authors":["Xin Zou","Yizhou Wang","Yibo Yan","Yuanhuiyi Lyu","Kening Zheng","Sirui Huang","Junkai Chen","Peijie Jiang","Jia Liu","Chang Tang","Xuming Hu"],"abstract":"Despite their impressive capabilities, multimodal large language models (MLLMs) are prone to hallucinations, i.e., the generated content that is nonsensical or unfaithful to input sources. Unlike in LLMs, hallucinations in MLLMs often stem from the sensitivity of text decoder to visual tokens, leading to a phenomenon akin to \"amnesia\" about visual information. To address this issue, we propose MemVR, a novel decoding paradigm inspired by common cognition: when the memory of an image seen the moment before is forgotten, people will look at it again for factual answers. Following this principle, we treat visual tokens as supplementary evidence, re-injecting them into the MLLM through Feed Forward Network (FFN) as \"key-value memory\" at the middle trigger layer. This \"look-twice\" mechanism occurs when the model exhibits high uncertainty during inference, effectively enhancing factual alignment. Comprehensive experimental evaluations demonstrate that MemVR significantly mitigates hallucination across various MLLMs and excels in general benchmarks without incurring additional time overhead. The implementation is available from https://github.com/1zhou-Wang/MemVR","url_abs":"https://arxiv.org/abs/2410.03577v2","url_pdf":"https://arxiv.org/pdf/2410.03577v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"look-twice-before-you-answer-memory-space","repo_url":"https://github.com/1zhou-Wang/MemVR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"hallucination","task_name":"Hallucination"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.03577","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.03577"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/1zhou-Wang/MemVR","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_draft_wrong":4,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1cadbcf04f8bc964","entry":"apply_rotary_pos_emb","repo":"1zhou-Wang/MemVR","repo_kind":"official","path":"modeling/modeling_chatglm.py","file_url":"https://github.com/1zhou-Wang/MemVR/blob/HEAD/modeling/modeling_chatglm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1cadbcf04f8bc964"}},{"code_sha256_prefix":"20e4f665698a3d18","entry":"collate_fn","repo":"1zhou-Wang/MemVR","repo_kind":"official","path":"eval/llava_model_vqa_loader.py","file_url":"https://github.com/1zhou-Wang/MemVR/blob/HEAD/eval/llava_model_vqa_loader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20e4f665698a3d18"}},{"code_sha256_prefix":"f5ff8b7846971cf9","entry":"collate_fn","repo":"1zhou-Wang/MemVR","repo_kind":"official","path":"eval/glm_model_vqa_loader.py","file_url":"https://github.com/1zhou-Wang/MemVR/blob/HEAD/eval/glm_model_vqa_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f5ff8b7846971cf9"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"1zhou-Wang/MemVR","repo_kind":"official","path":"eval/glm_model_vqa.py","file_url":"https://github.com/1zhou-Wang/MemVR/blob/HEAD/eval/glm_model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"1zhou-Wang/MemVR","repo_kind":"official","path":"eval/glm_model_vqa.py","file_url":"https://github.com/1zhou-Wang/MemVR/blob/HEAD/eval/glm_model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"d7e378ee7899fbde","entry":"split_tensor_along_last_dim","repo":"1zhou-Wang/MemVR","repo_kind":"official","path":"modeling/modeling_chatglm.py","file_url":"https://github.com/1zhou-Wang/MemVR/blob/HEAD/modeling/modeling_chatglm.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d7e378ee7899fbde"}},{"code_sha256_prefix":"237019e36b9481d9","entry":"default_init","repo":"1zhou-Wang/MemVR","repo_kind":"official","path":"modeling/modeling_chatglm.py","file_url":"https://github.com/1zhou-Wang/MemVR/blob/HEAD/modeling/modeling_chatglm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"237019e36b9481d9"}},{"code_sha256_prefix":"eb84abf07fedcf77","entry":"forward","repo":"1zhou-Wang/MemVR","repo_kind":"official","path":"memvr.py","file_url":"https://github.com/1zhou-Wang/MemVR/blob/HEAD/memvr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eb84abf07fedcf77"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}