{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/conflictbank-a-benchmark-for-evaluating-the","title":"ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM","arxiv_id":"2408.12076","date":"2024-08-22","proceeding":null,"authors":["Zhaochen Su","Jun Zhang","Xiaoye Qu","Tong Zhu","Yanshu Li","Jiashuo Sun","Juntao Li","Min Zhang","Yu Cheng"],"abstract":"Large language models (LLMs) have achieved impressive advancements across numerous disciplines, yet the critical issue of knowledge conflicts, a major source of hallucinations, has rarely been studied. Only a few research explored the conflicts between the inherent knowledge of LLMs and the retrieved contextual knowledge. However, a thorough assessment of knowledge conflict in LLMs is still missing. Motivated by this research gap, we present ConflictBank, the first comprehensive benchmark developed to systematically evaluate knowledge conflicts from three aspects: (i) conflicts encountered in retrieved knowledge, (ii) conflicts within the models' encoded knowledge, and (iii) the interplay between these conflict forms. Our investigation delves into four model families and twelve LLM instances, meticulously analyzing conflicts stemming from misinformation, temporal discrepancies, and semantic divergences. Based on our proposed novel construction framework, we create 7,453,853 claim-evidence pairs and 553,117 QA pairs. We present numerous findings on model scale, conflict causes, and conflict types. We hope our ConflictBank benchmark will help the community better understand model behavior in conflicts and develop more reliable LLMs.","url_abs":"https://arxiv.org/abs/2408.12076v1","url_pdf":"https://arxiv.org/pdf/2408.12076v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"conflictbank-a-benchmark-for-evaluating-the","repo_url":"https://github.com/zhaochen0110/conflictbank","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"misinformation","task_name":"Misinformation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.12076","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.12076"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhaochen0110/conflictbank","reach":{"status":"ok"}}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"93f55e3447c2ae71","entry":"calculate_accuracy","repo":"zhaochen0110/conflictbank","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/zhaochen0110/conflictbank/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93f55e3447c2ae71"}},{"code_sha256_prefix":"68dfae5ca75de5c4","entry":"calculate_entropy","repo":"zhaochen0110/conflictbank","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/zhaochen0110/conflictbank/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68dfae5ca75de5c4"}},{"code_sha256_prefix":"fe873fe963cdbc24","entry":"get_last_processed_index","repo":"zhaochen0110/conflictbank","repo_kind":"official","path":"generate_description.py","file_url":"https://github.com/zhaochen0110/conflictbank/blob/HEAD/generate_description.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe873fe963cdbc24"}},{"code_sha256_prefix":"814f8de63878efaf","entry":"is_english","repo":"zhaochen0110/conflictbank","repo_kind":"official","path":"data_construct.py","file_url":"https://github.com/zhaochen0110/conflictbank/blob/HEAD/data_construct.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"814f8de63878efaf"}},{"code_sha256_prefix":"badeb49663280ccc","entry":"mean_pooling","repo":"zhaochen0110/conflictbank","repo_kind":"official","path":"quality_control.py","file_url":"https://github.com/zhaochen0110/conflictbank/blob/HEAD/quality_control.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"badeb49663280ccc"}},{"code_sha256_prefix":"5c966b8355c69c08","entry":"process_json_file","repo":"zhaochen0110/conflictbank","repo_kind":"official","path":"evaluate.py","file_url":"https://github.com/zhaochen0110/conflictbank/blob/HEAD/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c966b8355c69c08"}},{"code_sha256_prefix":"9c4f7ad0932860f0","entry":"read_arrow_to_df_julia_ok","repo":"zhaochen0110/conflictbank","repo_kind":"official","path":"quality_control.py","file_url":"https://github.com/zhaochen0110/conflictbank/blob/HEAD/quality_control.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9c4f7ad0932860f0"}},{"code_sha256_prefix":"87dee3b4f453fe66","entry":"truncate_incomplete_sentence","repo":"zhaochen0110/conflictbank","repo_kind":"official","path":"quality_control.py","file_url":"https://github.com/zhaochen0110/conflictbank/blob/HEAD/quality_control.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87dee3b4f453fe66"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}