{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efuf-efficient-fine-grained-unlearning","title":"EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models","arxiv_id":"2402.09801","date":"2024-02-15","proceeding":null,"authors":["Shangyu Xing","Fei Zhao","Zhen Wu","Tuo An","WeiHao Chen","Chunhui Li","Jianbing Zhang","Xinyu Dai"],"abstract":"Multimodal large language models (MLLMs) have attracted increasing attention in the past few years, but they may still generate descriptions that include objects not present in the corresponding images, a phenomenon known as object hallucination. To eliminate hallucinations, existing methods manually annotate paired responses with and without hallucinations, and then employ various alignment algorithms to improve the alignment capability between images and text. However, they not only demand considerable computation resources during the finetuning stage but also require expensive human annotation to construct paired data needed by the alignment algorithms. To address these issues, we borrow the idea of unlearning and propose an efficient fine-grained unlearning framework (EFUF), which can eliminate hallucinations without the need for paired data. Extensive experiments show that our method consistently reduces hallucinations while preserving the generation quality with modest computational overhead. Our code and datasets will be publicly available.","url_abs":"https://arxiv.org/abs/2402.09801v3","url_pdf":"https://arxiv.org/pdf/2402.09801v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efuf-efficient-fine-grained-unlearning","repo_url":"https://github.com/starreeze/efuf","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"object-hallucination","task_name":"Object Hallucination"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.09801","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.09801"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/starreeze/efuf","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran":4,"ran_fixture":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"1a531e3c0d1094fb","entry":"bloom_forward","repo":"starreeze/efuf","repo_kind":"official","path":"Owl/mplug_owl/modeling_mplug_owl.py","file_url":"https://github.com/starreeze/efuf/blob/HEAD/Owl/mplug_owl/modeling_mplug_owl.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1a531e3c0d1094fb"}},{"code_sha256_prefix":"45567e61fe09093a","entry":"get_ltor_masks_and_position_ids_from_embeddings","repo":"starreeze/efuf","repo_kind":"official","path":"Owl/mplug_owl/modeling_mplug_owl.py","file_url":"https://github.com/starreeze/efuf/blob/HEAD/Owl/mplug_owl/modeling_mplug_owl.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"45567e61fe09093a"}},{"code_sha256_prefix":"265f4d55ccd60d18","entry":"get_media_indices","repo":"starreeze/efuf","repo_kind":"official","path":"Owl/mplug_owl/modeling_mplug_owl.py","file_url":"https://github.com/starreeze/efuf/blob/HEAD/Owl/mplug_owl/modeling_mplug_owl.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"265f4d55ccd60d18"}},{"code_sha256_prefix":"752b84a6f01c43d8","entry":"get_media_types","repo":"starreeze/efuf","repo_kind":"official","path":"Owl/mplug_owl_video/modeling_mplug_owl.py","file_url":"https://github.com/starreeze/efuf/blob/HEAD/Owl/mplug_owl_video/modeling_mplug_owl.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"752b84a6f01c43d8"}},{"code_sha256_prefix":"15e54d5d53ab90ac","entry":"post_process_code","repo":"starreeze/efuf","repo_kind":"official","path":"Owl/serve/model_utils.py","file_url":"https://github.com/starreeze/efuf/blob/HEAD/Owl/serve/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"15e54d5d53ab90ac"}},{"code_sha256_prefix":"607a861da4671ce5","entry":"post_process_output","repo":"starreeze/efuf","repo_kind":"official","path":"Owl/serve/model_utils.py","file_url":"https://github.com/starreeze/efuf/blob/HEAD/Owl/serve/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"607a861da4671ce5"}},{"code_sha256_prefix":"a12935f8894cc5de","entry":"gpt_infer","repo":"starreeze/efuf","repo_kind":"official","path":"common/gpt.py","file_url":"https://github.com/starreeze/efuf/blob/HEAD/common/gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a12935f8894cc5de"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}