{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unraveling-cross-modality-knowledge-conflict","title":"Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models","arxiv_id":"2410.03659","date":"2024-10-04","proceeding":null,"authors":["Tinghui Zhu","Qin Liu","Fei Wang","Zhengzhong Tu","Muhao Chen"],"abstract":"Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities for capturing and reasoning over multimodal inputs. However, these models are prone to parametric knowledge conflicts, which arise from inconsistencies of represented knowledge between their vision and language components. In this paper, we formally define the problem of $\\textbf{cross-modality parametric knowledge conflict}$ and present a systematic approach to detect, interpret, and mitigate them. We introduce a pipeline that identifies conflicts between visual and textual answers, showing a persistently high conflict rate across modalities in recent LVLMs regardless of the model size. We further investigate how these conflicts interfere with the inference process and propose a contrastive metric to discern the conflicting samples from the others. Building on these insights, we develop a novel dynamic contrastive decoding method that removes undesirable logits inferred from the less confident modality components based on answer confidence. For models that do not provide logits, we also introduce two prompt-based strategies to mitigate the conflicts. Our methods achieve promising improvements in accuracy on both the ViQuAE and InfoSeek datasets. Specifically, using LLaVA-34B, our proposed dynamic contrastive decoding improves an average accuracy of 2.24%.","url_abs":"https://arxiv.org/abs/2410.03659v2","url_pdf":"https://arxiv.org/pdf/2410.03659v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unraveling-cross-modality-knowledge-conflict","repo_url":"https://github.com/luka-group/vlm-knowledge-conflict","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.03659","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.03659"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/luka-group/vlm-knowledge-conflict","reach":null}],"summary":{"ran_draft_wrong":3,"ran_violates":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"6f996660f16793e2","entry":"clean_answer","repo":"luka-group/vlm-knowledge-conflict","repo_kind":"official","path":"src/analysis/post_hoc_contrastive_decoding_metric.py","file_url":"https://github.com/luka-group/vlm-knowledge-conflict/blob/HEAD/src/analysis/post_hoc_contrastive_decoding_metric.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f996660f16793e2"}},{"code_sha256_prefix":"d60fdb543dd96366","entry":"clean_answer","repo":"luka-group/vlm-knowledge-conflict","repo_kind":"official","path":"src/evaluate/evaluate_mc.py","file_url":"https://github.com/luka-group/vlm-knowledge-conflict/blob/HEAD/src/evaluate/evaluate_mc.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d60fdb543dd96366"}},{"code_sha256_prefix":"bcc5b7e1c3c18f93","entry":"match","repo":"luka-group/vlm-knowledge-conflict","repo_kind":"official","path":"src/evaluate/evaluate_mc.py","file_url":"https://github.com/luka-group/vlm-knowledge-conflict/blob/HEAD/src/evaluate/evaluate_mc.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bcc5b7e1c3c18f93"}},{"code_sha256_prefix":"cb804cc400ebe877","entry":"read_pred_file","repo":"luka-group/vlm-knowledge-conflict","repo_kind":"official","path":"src/evaluate/evaluate_mc.py","file_url":"https://github.com/luka-group/vlm-knowledge-conflict/blob/HEAD/src/evaluate/evaluate_mc.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb804cc400ebe877"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}