{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/refchecker-reference-based-fine-grained","title":"RefChecker: Reference-based Fine-grained Hallucination Checker and Benchmark for Large Language Models","arxiv_id":"2405.14486","date":"2024-05-23","proceeding":null,"authors":["Xiangkun Hu","Dongyu Ru","Lin Qiu","Qipeng Guo","Tianhang Zhang","Yang Xu","Yun Luo","PengFei Liu","Yue Zhang","Zheng Zhang"],"abstract":"Large Language Models (LLMs) have shown impressive capabilities but also a concerning tendency to hallucinate. This paper presents RefChecker, a framework that introduces claim-triplets to represent claims in LLM responses, aiming to detect fine-grained hallucinations. In RefChecker, an extractor generates claim-triplets from a response, which are then evaluated by a checker against a reference. We delineate three task settings: Zero, Noisy and Accurate Context, to reflect various real-world use cases. We curated a benchmark spanning various NLP tasks and annotated 11k claim-triplets from 2.1k responses by seven LLMs. RefChecker supports both proprietary and open-source models as the extractor and checker. Experiments demonstrate that claim-triplets enable superior hallucination detection, compared to other granularities such as response, sentence and sub-sentence level claims. RefChecker outperforms prior methods by 6.8 to 26.1 points on our benchmark and the checking results of RefChecker are strongly aligned with human judgments. This work is open sourced at https://github.com/amazon-science/RefChecker","url_abs":"https://arxiv.org/abs/2405.14486v1","url_pdf":"https://arxiv.org/pdf/2405.14486v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"refchecker-reference-based-fine-grained","repo_url":"https://github.com/amazon-science/refchecker","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"refchecker-reference-based-fine-grained","repo_url":"https://github.com/amazon-science/bschecker-for-fine-grained-hallucination-detection","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.14486","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.14486"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-science/bschecker-for-fine-grained-hallucination-detection","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-science/refchecker","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d3097c7b9c699684","entry":"get_evaluation_results","repo":"amazon-science/refchecker","repo_kind":"official","path":"benchmark/evaluation/evaluate.py","file_url":"https://github.com/amazon-science/refchecker/blob/HEAD/benchmark/evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d3097c7b9c699684"}},{"code_sha256_prefix":"725ceaa804cc0ce2","entry":"get_model_batch_response","repo":"amazon-science/bschecker-for-fine-grained-hallucination-detection","repo_kind":"listed","path":"refchecker/utils.py","file_url":"https://github.com/amazon-science/bschecker-for-fine-grained-hallucination-detection/blob/HEAD/refchecker/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"725ceaa804cc0ce2"}},{"code_sha256_prefix":"bfc76df2240c11f8","entry":"major_agg","repo":"amazon-science/bschecker-for-fine-grained-hallucination-detection","repo_kind":"listed","path":"refchecker/aggregator.py","file_url":"https://github.com/amazon-science/bschecker-for-fine-grained-hallucination-detection/blob/HEAD/refchecker/aggregator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bfc76df2240c11f8"}},{"code_sha256_prefix":"a7f923d5e6312cf4","entry":"soft_agg","repo":"amazon-science/bschecker-for-fine-grained-hallucination-detection","repo_kind":"listed","path":"refchecker/aggregator.py","file_url":"https://github.com/amazon-science/bschecker-for-fine-grained-hallucination-detection/blob/HEAD/refchecker/aggregator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7f923d5e6312cf4"}},{"code_sha256_prefix":"1ebc5b5aec806add","entry":"strict_agg","repo":"amazon-science/bschecker-for-fine-grained-hallucination-detection","repo_kind":"listed","path":"refchecker/aggregator.py","file_url":"https://github.com/amazon-science/bschecker-for-fine-grained-hallucination-detection/blob/HEAD/refchecker/aggregator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ebc5b5aec806add"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}