{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/factuality-enhanced-language-models-for-open","title":"Factuality Enhanced Language Models for Open-Ended Text Generation","arxiv_id":"2206.04624","date":"2022-06-09","proceeding":null,"authors":["Nayeon Lee","Wei Ping","Peng Xu","Mostofa Patwary","Pascale Fung","Mohammad Shoeybi","Bryan Catanzaro"],"abstract":"Pretrained language models (LMs) are susceptible to generate text with nonfactual information. In this work, we measure and improve the factual accuracy of large-scale LMs for open-ended text generation. We design the FactualityPrompts test set and metrics to measure the factuality of LM generations. Based on that, we study the factual accuracy of LMs with parameter sizes ranging from 126M to 530B. Interestingly, we find that larger LMs are more factual than smaller ones, although a previous study suggests that larger LMs can be less truthful in terms of misconceptions. In addition, popular sampling algorithms (e.g., top-p) in open-ended text generation can harm the factuality due to the ''uniform randomness'' introduced at every sampling step. We propose the factual-nucleus sampling algorithm that dynamically adapts the randomness to improve the factuality of generation while maintaining quality. Furthermore, we analyze the inefficiencies of the standard training method in learning correct associations between entities from factual text corpus (e.g., Wikipedia). We propose a factuality-enhanced training method that uses TopicPrefix for better awareness of facts and sentence completion as the training objective, which can vastly reduce the factual errors. We release our code and FactualityPrompts benchmark at: https://github.com/nayeon7lee/FactualityPrompt.","url_abs":"https://arxiv.org/abs/2206.04624v3","url_pdf":"https://arxiv.org/pdf/2206.04624v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"factuality-enhanced-language-models-for-open","repo_url":"https://github.com/nayeon7lee/factualityprompt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"factuality-enhanced-language-models-for-open","repo_url":"https://github.com/NVIDIA/FasterTransformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"factuality-enhanced-language-models-for-open","repo_url":"https://github.com/niyunsheng/ems-sd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"factuality-enhanced-language-models-for-open","repo_url":"https://github.com/ranggihwang/pregated_moe","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"factuality-enhanced-language-models-for-open","repo_url":"https://github.com/triton-inference-server/fastertransformer_backend","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"misconceptions","task_name":"Misconceptions"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-completion","task_name":"Sentence Completion"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.04624","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.04624"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/nayeon7lee/FactualityPrompt","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ranggihwang/pregated_moe","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NVIDIA/FasterTransformer","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nayeon7lee/factualityprompt","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/niyunsheng/ems-sd","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/triton-inference-server/fastertransformer_backend","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":9},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1},"listed":{"samples":5,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"ae1cfea3ae5b4973","entry":"has_incorrect_style","repo":"nayeon7lee/FactualityPrompt","repo_kind":"official","path":"src/claim_handling.py","file_url":"https://github.com/nayeon7lee/FactualityPrompt/blob/HEAD/src/claim_handling.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ae1cfea3ae5b4973"}},{"code_sha256_prefix":"77f69af078e09830","entry":"mean_pooling","repo":"nayeon7lee/FactualityPrompt","repo_kind":"official","path":"src/retriever.py","file_url":"https://github.com/nayeon7lee/FactualityPrompt/blob/HEAD/src/retriever.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77f69af078e09830"}},{"code_sha256_prefix":"64e26deeb22d1c2c","entry":"clean_wiki_sents","repo":"nayeon7lee/FactualityPrompt","repo_kind":"official","path":"src/retriever.py","file_url":"https://github.com/nayeon7lee/FactualityPrompt/blob/HEAD/src/retriever.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"64e26deeb22d1c2c"}},{"code_sha256_prefix":"d6e6d96f11cb2f7d","entry":"create_dependencies","repo":"triton-inference-server/fastertransformer_backend","repo_kind":"listed","path":"docker/create_dockerfile_and_build.py","file_url":"https://github.com/triton-inference-server/fastertransformer_backend/blob/HEAD/docker/create_dockerfile_and_build.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"d6e6d96f11cb2f7d"}},{"code_sha256_prefix":"e05fa8115f8f0e89","entry":"create_postbuild","repo":"triton-inference-server/fastertransformer_backend","repo_kind":"listed","path":"docker/create_dockerfile_and_build.py","file_url":"https://github.com/triton-inference-server/fastertransformer_backend/blob/HEAD/docker/create_dockerfile_and_build.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"e05fa8115f8f0e89"}},{"code_sha256_prefix":"9efefacf952612d1","entry":"modify_logits_for_top_p_filtering","repo":"ranggihwang/pregated_moe","repo_kind":"listed","path":"3rdparty/Megatron-LM/megatron/text_generation/sampling.py","file_url":"https://github.com/ranggihwang/pregated_moe/blob/HEAD/3rdparty/Megatron-LM/megatron/text_generation/sampling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9efefacf952612d1"}},{"code_sha256_prefix":"b204d507cc3d48d6","entry":"ner_metric","repo":"nayeon7lee/FactualityPrompt","repo_kind":"official","path":"src/factuality_metric.py","file_url":"https://github.com/nayeon7lee/FactualityPrompt/blob/HEAD/src/factuality_metric.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b204d507cc3d48d6"}},{"code_sha256_prefix":"65dbeb9236d5226e","entry":"preprocess","repo":"nayeon7lee/FactualityPrompt","repo_kind":"official","path":"fever_athene/src/retrieval/filter_lists.py","file_url":"https://github.com/nayeon7lee/FactualityPrompt/blob/HEAD/fever_athene/src/retrieval/filter_lists.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"65dbeb9236d5226e"}},{"code_sha256_prefix":"1627a3eb7ca6505e","entry":"to_word_list_format","repo":"triton-inference-server/fastertransformer_backend","repo_kind":"listed","path":"all_models/bloom/preprocessing/1/model.py","file_url":"https://github.com/triton-inference-server/fastertransformer_backend/blob/HEAD/all_models/bloom/preprocessing/1/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"1627a3eb7ca6505e"}},{"code_sha256_prefix":"e12f122ceb28fd4b","entry":"to_word_list_format","repo":"triton-inference-server/fastertransformer_backend","repo_kind":"listed","path":"all_models/gptj/preprocessing/1/word_list.py","file_url":"https://github.com/triton-inference-server/fastertransformer_backend/blob/HEAD/all_models/gptj/preprocessing/1/word_list.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":false,"mcp_get_code":{"code_sha256":"e12f122ceb28fd4b"}},{"code_sha256_prefix":"1e65673973a229c2","entry":"uninformative","repo":"nayeon7lee/FactualityPrompt","repo_kind":"official","path":"fever_athene/src/retrieval/filter_lists.py","file_url":"https://github.com/nayeon7lee/FactualityPrompt/blob/HEAD/fever_athene/src/retrieval/filter_lists.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e65673973a229c2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}