{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-new-benchmark-and-reverse-validation-method","title":"A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection","arxiv_id":"2310.06498","date":"2023-10-10","proceeding":null,"authors":["Shiping Yang","Renliang Sun","Xiaojun Wan"],"abstract":"Large Language Models (LLMs) have shown their ability to collaborate effectively with humans in real-world scenarios. However, LLMs are apt to generate hallucinations, i.e., makeup incorrect text and unverified information, which can cause significant damage when deployed for mission-critical tasks. In this paper, we propose a self-check approach based on reverse validation to detect factual errors automatically in a zero-resource fashion. To facilitate future studies and assess different methods, we construct a hallucination detection benchmark named PHD, which is generated by ChatGPT and annotated by human annotators. Contrasting previous studies of zero-resource hallucination detection, our method and benchmark concentrate on passage-level detection instead of sentence-level. We empirically evaluate our method and existing zero-resource detection methods on two datasets. The experimental results demonstrate that the proposed method considerably outperforms the baselines while costing fewer tokens and less time. Furthermore, we manually analyze some hallucination cases that LLM failed to capture, revealing the shared limitation of zero-resource methods.","url_abs":"https://arxiv.org/abs/2310.06498v2","url_pdf":"https://arxiv.org/pdf/2310.06498v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-new-benchmark-and-reverse-validation-method","repo_url":"https://github.com/maybenotime/phd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.06498","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.06498"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/maybenotime/phd","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"b4c052fb373aef07","entry":"load_llama_from_config","repo":"maybenotime/phd","repo_kind":"official","path":"Ablation_Study_Llama2-7b/inference/model_utils.py","file_url":"https://github.com/maybenotime/phd/blob/HEAD/Ablation_Study_Llama2-7b/inference/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4c052fb373aef07"}},{"code_sha256_prefix":"6d5ba8546b6aad2c","entry":"load_peft_model","repo":"maybenotime/phd","repo_kind":"official","path":"Ablation_Study_Llama2-7b/inference/model_utils.py","file_url":"https://github.com/maybenotime/phd/blob/HEAD/Ablation_Study_Llama2-7b/inference/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d5ba8546b6aad2c"}},{"code_sha256_prefix":"306222c62669fec0","entry":"request_api","repo":"maybenotime/phd","repo_kind":"official","path":"SelfCheckBERTScore/sample_PHD.py","file_url":"https://github.com/maybenotime/phd/blob/HEAD/SelfCheckBERTScore/sample_PHD.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"306222c62669fec0"}},{"code_sha256_prefix":"4b9c99e805944c4e","entry":"detect_hal","repo":"maybenotime/phd","repo_kind":"official","path":"LMvsLM_replicate/LMvsLM_detect.py","file_url":"https://github.com/maybenotime/phd/blob/HEAD/LMvsLM_replicate/LMvsLM_detect.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b9c99e805944c4e"}},{"code_sha256_prefix":"c7366fd47611b08f","entry":"load_model","repo":"maybenotime/phd","repo_kind":"official","path":"Ablation_Study_Llama2-7b/inference/model_utils.py","file_url":"https://github.com/maybenotime/phd/blob/HEAD/Ablation_Study_Llama2-7b/inference/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7366fd47611b08f"}},{"code_sha256_prefix":"0b65abd27e708564","entry":"samples","repo":"maybenotime/phd","repo_kind":"official","path":"SelfCheckBERTScore/sample_PHD.py","file_url":"https://github.com/maybenotime/phd/blob/HEAD/SelfCheckBERTScore/sample_PHD.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b65abd27e708564"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}