{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unveiling-the-siren-s-song-towards-reliable","title":"FactCHD: Benchmarking Fact-Conflicting Hallucination Detection","arxiv_id":"2310.12086","date":"2023-10-18","proceeding":null,"authors":["Xiang Chen","Duanzheng Song","Honghao Gui","Chenxi Wang","Ningyu Zhang","Yong Jiang","Fei Huang","Chengfei Lv","Dan Zhang","Huajun Chen"],"abstract":"Despite their impressive generative capabilities, LLMs are hindered by fact-conflicting hallucinations in real-world applications. The accurate identification of hallucinations in texts generated by LLMs, especially in complex inferential scenarios, is a relatively unexplored area. To address this gap, we present FactCHD, a dedicated benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. A distinctive element of FactCHD is its integration of fact-based evidence chains, significantly enhancing the depth of evaluating the detectors' explanations. Experiments on different LLMs expose the shortcomings of current approaches in detecting factual errors accurately. Furthermore, we introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2, aiming to yield more credible detection through the amalgamation of predictive results and evidence. The benchmark dataset is available at https://github.com/zjunlp/FactCHD.","url_abs":"https://arxiv.org/abs/2310.12086v3","url_pdf":"https://arxiv.org/pdf/2310.12086v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unveiling-the-siren-s-song-towards-reliable","repo_url":"https://github.com/zjunlp/factchd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"hallucination","task_name":"Hallucination"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.12086","atlas_url":"https://app.syntology.ai/?focus=2310.12086","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.12086"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zjunlp/factchd","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"de93f7b351850ef0","entry":"random_select_n_from_list","repo":"zjunlp/factchd","repo_kind":"official","path":"data_generate/kg_data_generate.py","file_url":"https://github.com/zjunlp/factchd/blob/HEAD/data_generate/kg_data_generate.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de93f7b351850ef0"}},{"code_sha256_prefix":"399d40d563633e82","entry":"set_default","repo":"zjunlp/factchd","repo_kind":"official","path":"data_generate/kg_data_generate.py","file_url":"https://github.com/zjunlp/factchd/blob/HEAD/data_generate/kg_data_generate.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"399d40d563633e82"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}