{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/halle-switch-rethinking-and-controlling","title":"HallE-Control: Controlling Object Hallucination in Large Multimodal Models","arxiv_id":"2310.01779","date":"2023-10-03","proceeding":null,"authors":["Bohan Zhai","Shijia Yang","Chenfeng Xu","Sheng Shen","Kurt Keutzer","Chunyuan Li","Manling Li"],"abstract":"Current Large Multimodal Models (LMMs) achieve remarkable progress, yet there remains significant uncertainty regarding their ability to accurately apprehend visual details, that is, in performing detailed captioning. To address this, we introduce $\\textit{CCEval}$, a GPT-4 assisted evaluation method for detailed captioning. Interestingly, while LMMs demonstrate minimal object existence hallucination in existing VQA benchmarks, our proposed evaluation reveals continued susceptibility to such hallucinations. In this paper, we make the first attempt to investigate such hallucination from different aspects, including image resolution, the language decoder size, and instruction data amount, quality, granularity. Our findings underscore the unwarranted inference when the language description includes details at a finer object granularity than what the vision module can ground or verify, thus inducing hallucination. To control such hallucinations, we further attribute the reliability of captioning to contextual knowledge (involving only contextually grounded objects) and parametric knowledge (containing inferred objects by the model). Thus, we introduce $\\textit{HallE-Control}$, a controllable LMM in terms of $\\textbf{Hall}$ucination in object $\\textbf{E}$xistence. HallE-Control can condition the captioning to shift between (i) exclusively depicting contextual knowledge for grounded objects and (ii) blending it with parametric knowledge to imagine inferred objects. Our method reduces hallucination by 44% compared to LLaVA$_{7B}$ and maintains the object coverage.","url_abs":"https://arxiv.org/abs/2310.01779v3","url_pdf":"https://arxiv.org/pdf/2310.01779v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"halle-switch-rethinking-and-controlling","repo_url":"https://github.com/bronyayang/HallE_Switch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"halle-switch-rethinking-and-controlling","repo_url":"https://github.com/bronyayang/halle_control","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-hallucination","task_name":"Object Hallucination"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.01779","atlas_url":"https://app.syntology.ai/?focus=2310.01779","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.01779"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bronyayang/halle_control","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bronyayang/HallE_Switch","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"ran_honours":1,"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":9,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"592b3c1a88f93d7c","entry":"expand2square","repo":"bronyayang/HallE_Switch","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/bronyayang/HallE_Switch/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"592b3c1a88f93d7c"}},{"code_sha256_prefix":"0adf9e0153200aed","entry":"load_generated_captions","repo":"bronyayang/halle_control","repo_kind":"official","path":"cceval.py","file_url":"https://github.com/bronyayang/halle_control/blob/HEAD/cceval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0adf9e0153200aed"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"bronyayang/HallE_Switch","repo_kind":"official","path":"eval/model_controller.py","file_url":"https://github.com/bronyayang/HallE_Switch/blob/HEAD/eval/model_controller.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"c3ee9d07c900dd55","entry":"load_image_from_base64","repo":"bronyayang/HallE_Switch","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/bronyayang/HallE_Switch/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3ee9d07c900dd55"}},{"code_sha256_prefix":"1e1fb53310bf974a","entry":"print_metrics","repo":"bronyayang/halle_control","repo_kind":"official","path":"cceval.py","file_url":"https://github.com/bronyayang/halle_control/blob/HEAD/cceval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e1fb53310bf974a"}},{"code_sha256_prefix":"fa1225dfac92bc0d","entry":"get_peft_state_maybe_zero_3","repo":"bronyayang/HallE_Switch","repo_kind":"official","path":"llava/train/train_switch.py","file_url":"https://github.com/bronyayang/HallE_Switch/blob/HEAD/llava/train/train_switch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa1225dfac92bc0d"}},{"code_sha256_prefix":"1c53657305b66e9f","entry":"get_peft_state_non_lora_maybe_zero_3","repo":"bronyayang/HallE_Switch","repo_kind":"official","path":"llava/train/train_switch.py","file_url":"https://github.com/bronyayang/HallE_Switch/blob/HEAD/llava/train/train_switch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c53657305b66e9f"}},{"code_sha256_prefix":"616ffbdc154ed2d8","entry":"maybe_zero_3","repo":"bronyayang/HallE_Switch","repo_kind":"official","path":"llava/train/train_switch.py","file_url":"https://github.com/bronyayang/HallE_Switch/blob/HEAD/llava/train/train_switch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"616ffbdc154ed2d8"}},{"code_sha256_prefix":"344dff4791fd1381","entry":"process_images","repo":"bronyayang/HallE_Switch","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/bronyayang/HallE_Switch/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"344dff4791fd1381"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}