{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hallucidoctor-mitigating-hallucinatory","title":"HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data","arxiv_id":"2311.13614","date":"2023-11-22","proceeding":"CVPR 2024 1","authors":["Qifan Yu","Juncheng Li","Longhui Wei","Liang Pang","Wentao Ye","Bosheng Qin","Siliang Tang","Qi Tian","Yueting Zhuang"],"abstract":"Multi-modal Large Language Models (MLLMs) tuned on machine-generated instruction-following data have demonstrated remarkable performance in various multi-modal understanding and generation tasks. However, the hallucinations inherent in machine-generated data, which could lead to hallucinatory outputs in MLLMs, remain under-explored. This work aims to investigate various hallucinations (i.e., object, relation, attribute hallucinations) and mitigate those hallucinatory toxicities in large-scale machine-generated visual instruction datasets. Drawing on the human ability to identify factual errors, we present a novel hallucination detection and elimination framework, HalluciDoctor, based on the cross-checking paradigm. We use our framework to identify and eliminate hallucinations in the training data automatically. Interestingly, HalluciDoctor also indicates that spurious correlations arising from long-tail object co-occurrences contribute to hallucinations. Based on that, we execute counterfactual visual instruction expansion to balance data distribution, thereby enhancing MLLMs' resistance to hallucinations. Comprehensive experiments on hallucination evaluation benchmarks show that our method successfully mitigates 44.6% hallucinations relatively and maintains competitive performance compared to LLaVA. The data and code for this paper are publicly available. \\url{https://github.com/Yuqifan1117/HalluciDoctor}.","url_abs":"https://arxiv.org/abs/2311.13614v2","url_pdf":"https://arxiv.org/pdf/2311.13614v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hallucidoctor-mitigating-hallucinatory","repo_url":"https://github.com/yuqifan1117/hallucidoctor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"hallucination-evaluation","task_name":"Hallucination Evaluation"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":null,"task_name":"counterfactual"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.13614","atlas_url":"https://app.syntology.ai/?focus=2311.13614","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.13614"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuqifan1117/hallucidoctor","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"d1ffbb6af3208715","entry":"format_inputs","repo":"yuqifan1117/hallucidoctor","repo_kind":"official","path":"models/blip2_candidate_answer_generator.py","file_url":"https://github.com/yuqifan1117/hallucidoctor/blob/HEAD/models/blip2_candidate_answer_generator.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1ffbb6af3208715"}},{"code_sha256_prefix":"c89f9d2c4123da91","entry":"cropimage2image","repo":"yuqifan1117/hallucidoctor","repo_kind":"official","path":"utils/cropimage2image.py","file_url":"https://github.com/yuqifan1117/hallucidoctor/blob/HEAD/utils/cropimage2image.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c89f9d2c4123da91"}},{"code_sha256_prefix":"823c8f3017c8e193","entry":"mllm_experts_load","repo":"yuqifan1117/hallucidoctor","repo_kind":"official","path":"models/blip2_candidate_answer_generator.py","file_url":"https://github.com/yuqifan1117/hallucidoctor/blob/HEAD/models/blip2_candidate_answer_generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"823c8f3017c8e193"}},{"code_sha256_prefix":"660ed1d7e213c2a3","entry":"multianswer_exact_match","repo":"yuqifan1117/hallucidoctor","repo_kind":"official","path":"models/evaluation_utils.py","file_url":"https://github.com/yuqifan1117/hallucidoctor/blob/HEAD/models/evaluation_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"660ed1d7e213c2a3"}},{"code_sha256_prefix":"56fd4c74dfbb98ac","entry":"preprocess_answer","repo":"yuqifan1117/hallucidoctor","repo_kind":"official","path":"models/evaluation_utils.py","file_url":"https://github.com/yuqifan1117/hallucidoctor/blob/HEAD/models/evaluation_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56fd4c74dfbb98ac"}},{"code_sha256_prefix":"2084871add13d2d4","entry":"singleanswer_exact_match","repo":"yuqifan1117/hallucidoctor","repo_kind":"official","path":"models/evaluation_utils.py","file_url":"https://github.com/yuqifan1117/hallucidoctor/blob/HEAD/models/evaluation_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2084871add13d2d4"}},{"code_sha256_prefix":"e966d99425c4d784","entry":"synset_list","repo":"yuqifan1117/hallucidoctor","repo_kind":"official","path":"models/consistency_crosscheck_object.py","file_url":"https://github.com/yuqifan1117/hallucidoctor/blob/HEAD/models/consistency_crosscheck_object.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e966d99425c4d784"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}