{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-curious-case-of-neural-text-degeneration","title":"The Curious Case of Neural Text Degeneration","arxiv_id":"1904.09751","date":"2019-04-22","proceeding":"ICLR 2020 1","authors":["Ari Holtzman","Jan Buys","Li Du","Maxwell Forbes","Yejin Choi"],"abstract":"Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.","url_abs":"https://arxiv.org/abs/1904.09751v2","url_pdf":"https://arxiv.org/pdf/1904.09751v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/ari-holtzman/degen","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/Advancing-Machine-Human-Reasoning-Lab/apt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/AleksSol/mmp_activity_diary","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/THUDM/KOBE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/THUcqb/KOBE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/TaiseiAso/BiGruAttEncDec","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/c00k1ez/plain-transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/colorfulscoop/tfdlg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/microsoft/kid","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/noriyukipy/tfchat","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/noriyukipy/tfdlg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/prakhar21/textaugmentation-gpt2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/qibinc/KOBE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/sander102907/autoencoder_program_synthesis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/thu-coai/CDial-GPT","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/MindSpore-scientific/code-8/tree/main/QuantExplainNLP_AlgoAnalysis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"the-curious-case-of-neural-text-degeneration","repo_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1904.09751","atlas_url":"https://app.syntology.ai/?focus=1904.09751","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1904.09751"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/prakhar21/textaugmentation-gpt2","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/THUcqb/KOBE","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AleksSol/mmp_activity_diary","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sander102907/autoencoder_program_synthesis","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/noriyukipy/tfchat","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ari-holtzman/degen","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/kid","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/THUDM/KOBE","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/colorfulscoop/tfdlg","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qibinc/KOBE","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/TaiseiAso/BiGruAttEncDec","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Advancing-Machine-Human-Reasoning-Lab/apt","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/c00k1ez/plain-transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thu-coai/CDial-GPT","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/noriyukipy/tfdlg","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-8/tree/main/QuantExplainNLP_AlgoAnalysis","reach":null}],"summary":{"ran":4,"ran_draft_wrong":4,"ran_honours":1,"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1},"listed":{"samples":10,"ran":5,"repositories":5}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"5a0768b2c98ccd3a","entry":"NucleusSampler","repo":"labmlai/annotated_deep_learning_paper_implementations","repo_kind":"listed","path":"labml_nn/sampling/nucleus.py","file_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/HEAD/labml_nn/sampling/nucleus.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a0768b2c98ccd3a"}},{"code_sha256_prefix":"27aeb9112fe4cf52","entry":"Sampling","repo":"sander102907/autoencoder_program_synthesis","repo_kind":"listed","path":"autoencoder_program_synthesis/utils/Sampling.py","file_url":"https://github.com/sander102907/autoencoder_program_synthesis/blob/HEAD/autoencoder_program_synthesis/utils/Sampling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"27aeb9112fe4cf52"}},{"code_sha256_prefix":"2d61025ad472d12d","entry":"_gumbel","repo":"ari-holtzman/degen","repo_kind":"official","path":"gen.py","file_url":"https://github.com/ari-holtzman/degen/blob/HEAD/gen.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2d61025ad472d12d"}},{"code_sha256_prefix":"7c5ca0a2e17c5e6b","entry":"_shift_gumbel_maximum","repo":"ari-holtzman/degen","repo_kind":"official","path":"gen.py","file_url":"https://github.com/ari-holtzman/degen/blob/HEAD/gen.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7c5ca0a2e17c5e6b"}},{"code_sha256_prefix":"6135894d997967b3","entry":"accuracy","repo":"THUDM/KOBE","repo_kind":"listed","path":"kobe/utils/helpers.py","file_url":"https://github.com/THUDM/KOBE/blob/HEAD/kobe/utils/helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6135894d997967b3"}},{"code_sha256_prefix":"f10aa6a3b19fd64e","entry":"filter_to_topp","repo":"noriyukipy/tfdlg","repo_kind":"listed","path":"tfdlg/generations.py","file_url":"https://github.com/noriyukipy/tfdlg/blob/HEAD/tfdlg/generations.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f10aa6a3b19fd64e"}},{"code_sha256_prefix":"b9ed4ea16c8288f0","entry":"get_bert_vocab_size","repo":"THUDM/KOBE","repo_kind":"listed","path":"kobe/utils/helpers.py","file_url":"https://github.com/THUDM/KOBE/blob/HEAD/kobe/utils/helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9ed4ea16c8288f0"}},{"code_sha256_prefix":"46d2256fd27f9ce4","entry":"gumbel_sbs_decode","repo":"ari-holtzman/degen","repo_kind":"official","path":"gen.py","file_url":"https://github.com/ari-holtzman/degen/blob/HEAD/gen.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"46d2256fd27f9ce4"}},{"code_sha256_prefix":"975e63af902daf2e","entry":"gumbel_with_maximum","repo":"ari-holtzman/degen","repo_kind":"official","path":"gen.py","file_url":"https://github.com/ari-holtzman/degen/blob/HEAD/gen.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"975e63af902daf2e"}},{"code_sha256_prefix":"695eaa24be6f2a1d","entry":"top_filtering","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"695eaa24be6f2a1d"}},{"code_sha256_prefix":"7d8fed5b92965553","entry":"Sampler","repo":"labmlai/annotated_deep_learning_paper_implementations","repo_kind":"listed","path":"labml_nn/sampling/nucleus.py","file_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/HEAD/labml_nn/sampling/nucleus.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d8fed5b92965553"}},{"code_sha256_prefix":"717c7692322af957","entry":"entity_enhance","repo":"TaiseiAso/BiGruAttEncDec","repo_kind":"listed","path":"decode.py","file_url":"https://github.com/TaiseiAso/BiGruAttEncDec/blob/HEAD/decode.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"717c7692322af957"}},{"code_sha256_prefix":"c8316a45e99146db","entry":"gumbel","repo":"ari-holtzman/degen","repo_kind":"official","path":"gen.py","file_url":"https://github.com/ari-holtzman/degen/blob/HEAD/gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c8316a45e99146db"}},{"code_sha256_prefix":"6c601d82e95610de","entry":"gumbel_like","repo":"ari-holtzman/degen","repo_kind":"official","path":"gen.py","file_url":"https://github.com/ari-holtzman/degen/blob/HEAD/gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6c601d82e95610de"}},{"code_sha256_prefix":"7b0560d1ecad407a","entry":"inf_suppression","repo":"TaiseiAso/BiGruAttEncDec","repo_kind":"listed","path":"decode.py","file_url":"https://github.com/TaiseiAso/BiGruAttEncDec/blob/HEAD/decode.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b0560d1ecad407a"}},{"code_sha256_prefix":"4aa0814a647dbccb","entry":"repetitive_suppression","repo":"TaiseiAso/BiGruAttEncDec","repo_kind":"listed","path":"decode.py","file_url":"https://github.com/TaiseiAso/BiGruAttEncDec/blob/HEAD/decode.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4aa0814a647dbccb"}},{"code_sha256_prefix":"2cd2fb7377190229","entry":"top_p_sampling_search","repo":"TaiseiAso/BiGruAttEncDec","repo_kind":"listed","path":"decode.py","file_url":"https://github.com/TaiseiAso/BiGruAttEncDec/blob/HEAD/decode.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2cd2fb7377190229"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}