{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/selfcheckgpt-zero-resource-black-box","title":"SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models","arxiv_id":"2303.08896","date":"2023-03-15","proceeding":null,"authors":["Potsawee Manakul","Adian Liusie","Mark J. F. Gales"],"abstract":"Generative Large Language Models (LLMs) such as GPT-3 are capable of generating highly fluent responses to a wide variety of user prompts. However, LLMs are known to hallucinate facts and make non-factual statements which can undermine trust in their output. Existing fact-checking approaches either require access to the output probability distribution (which may not be available for systems such as ChatGPT) or external databases that are interfaced via separate, often complex, modules. In this work, we propose \"SelfCheckGPT\", a simple sampling-based approach that can be used to fact-check the responses of black-box models in a zero-resource fashion, i.e. without an external database. SelfCheckGPT leverages the simple idea that if an LLM has knowledge of a given concept, sampled responses are likely to be similar and contain consistent facts. However, for hallucinated facts, stochastically sampled responses are likely to diverge and contradict one another. We investigate this approach by using GPT-3 to generate passages about individuals from the WikiBio dataset, and manually annotate the factuality of the generated passages. We demonstrate that SelfCheckGPT can: i) detect non-factual and factual sentences; and ii) rank passages in terms of factuality. We compare our approach to several baselines and show that our approach has considerably higher AUC-PR scores in sentence-level hallucination detection and higher correlation scores in passage-level factuality assessment compared to grey-box methods.","url_abs":"https://arxiv.org/abs/2303.08896v3","url_pdf":"https://arxiv.org/pdf/2303.08896v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"selfcheckgpt-zero-resource-black-box","repo_url":"https://github.com/potsawee/selfcheckgpt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"fact-checking","task_name":"Fact Checking"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[{"slug":"wikibio-gpt-3-hallucination-dataset","name":"WikiBio GPT-3 Hallucination Dataset","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.08896","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.08896"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/potsawee/selfcheckgpt","reach":null}],"summary":{"ran":5,"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":11,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"48303d9614eef740","entry":"method_bayes_with_alpha","repo":"potsawee/selfcheckgpt","repo_kind":"official","path":"selfcheckgpt/modeling_selfcheck.py","file_url":"https://github.com/potsawee/selfcheckgpt/blob/HEAD/selfcheckgpt/modeling_selfcheck.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48303d9614eef740"}},{"code_sha256_prefix":"a327c65c24bd1adc","entry":"method_simple_counting","repo":"potsawee/selfcheckgpt","repo_kind":"official","path":"selfcheckgpt/modeling_selfcheck.py","file_url":"https://github.com/potsawee/selfcheckgpt/blob/HEAD/selfcheckgpt/modeling_selfcheck.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a327c65c24bd1adc"}},{"code_sha256_prefix":"fd8b74b87c682b53","entry":"method_vanilla_bayes","repo":"potsawee/selfcheckgpt","repo_kind":"official","path":"selfcheckgpt/modeling_selfcheck.py","file_url":"https://github.com/potsawee/selfcheckgpt/blob/HEAD/selfcheckgpt/modeling_selfcheck.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd8b74b87c682b53"}},{"code_sha256_prefix":"91aa51bfaaa9af1c","entry":"prepare_distractor_input","repo":"potsawee/selfcheckgpt","repo_kind":"official","path":"selfcheckgpt/modeling_selfcheck.py","file_url":"https://github.com/potsawee/selfcheckgpt/blob/HEAD/selfcheckgpt/modeling_selfcheck.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91aa51bfaaa9af1c"}},{"code_sha256_prefix":"0f6a58be9f56b183","entry":"prepare_qa_input","repo":"potsawee/selfcheckgpt","repo_kind":"official","path":"selfcheckgpt/modeling_selfcheck.py","file_url":"https://github.com/potsawee/selfcheckgpt/blob/HEAD/selfcheckgpt/modeling_selfcheck.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f6a58be9f56b183"}},{"code_sha256_prefix":"8d22955c5597692a","entry":"question_generation_sentence_level","repo":"potsawee/selfcheckgpt","repo_kind":"official","path":"selfcheckgpt/modeling_selfcheck.py","file_url":"https://github.com/potsawee/selfcheckgpt/blob/HEAD/selfcheckgpt/modeling_selfcheck.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d22955c5597692a"}},{"code_sha256_prefix":"6f5cf383ae436397","entry":"MQAGConfig","repo":"potsawee/selfcheckgpt","repo_kind":"official","path":"selfcheckgpt/modeling_selfcheck.py","file_url":"https://github.com/potsawee/selfcheckgpt/blob/HEAD/selfcheckgpt/modeling_selfcheck.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f5cf383ae436397"}},{"code_sha256_prefix":"e976fc7d0ba93b93","entry":"SelfCheckMQAG","repo":"potsawee/selfcheckgpt","repo_kind":"official","path":"selfcheckgpt/modeling_selfcheck.py","file_url":"https://github.com/potsawee/selfcheckgpt/blob/HEAD/selfcheckgpt/modeling_selfcheck.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e976fc7d0ba93b93"}},{"code_sha256_prefix":"1cd98422f1661366","entry":"answerability_scoring","repo":"potsawee/selfcheckgpt","repo_kind":"official","path":"selfcheckgpt/modeling_selfcheck.py","file_url":"https://github.com/potsawee/selfcheckgpt/blob/HEAD/selfcheckgpt/modeling_selfcheck.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1cd98422f1661366"}},{"code_sha256_prefix":"2374d4bcf09066d3","entry":"answering","repo":"potsawee/selfcheckgpt","repo_kind":"official","path":"selfcheckgpt/modeling_selfcheck.py","file_url":"https://github.com/potsawee/selfcheckgpt/blob/HEAD/selfcheckgpt/modeling_selfcheck.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2374d4bcf09066d3"}},{"code_sha256_prefix":"b38e4eaa9c590ae7","entry":"prepare_answering_input","repo":"potsawee/selfcheckgpt","repo_kind":"official","path":"selfcheckgpt/modeling_selfcheck.py","file_url":"https://github.com/potsawee/selfcheckgpt/blob/HEAD/selfcheckgpt/modeling_selfcheck.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b38e4eaa9c590ae7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}