{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-text-generation-with-unlikelihood","title":"Neural Text Generation with Unlikelihood Training","arxiv_id":"1908.04319","date":"2019-08-12","proceeding":"ICLR 2020 1","authors":["Sean Welleck","Ilia Kulikov","Stephen Roller","Emily Dinan","Kyunghyun Cho","Jason Weston"],"abstract":"Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-$k$ and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques.","url_abs":"https://arxiv.org/abs/1908.04319v2","url_pdf":"https://arxiv.org/pdf/1908.04319v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-text-generation-with-unlikelihood","repo_url":"https://github.com/facebookresearch/unlikelihood_training","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"neural-text-generation-with-unlikelihood","repo_url":"https://github.com/c00k1ez/plain-transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"neural-text-generation-with-unlikelihood","repo_url":"https://github.com/fadedcosine/pos-guided-neural-text-generation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"neural-text-generation-with-unlikelihood","repo_url":"https://github.com/google/t5patches","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"neural-text-generation-with-unlikelihood","repo_url":"https://github.com/griff4692/calibrating-summaries","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"neural-text-generation-with-unlikelihood","repo_url":"https://github.com/ljyflores/loss-library","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"blocking","task_name":"Blocking"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1908.04319","atlas_url":"https://app.syntology.ai/?focus=1908.04319","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1908.04319"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google/t5patches","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fadedcosine/pos-guided-neural-text-generation","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/griff4692/calibrating-summaries","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/unlikelihood_training","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/c00k1ez/plain-transformers","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ljyflores/loss-library","reach":null}],"summary":{"ran":2,"ran_fixture":3,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":5,"ran":5,"repositories":4}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"785d35781789b4f4","entry":"CandidateLoss","repo":"fadedcosine/pos-guided-neural-text-generation","repo_kind":"listed","path":"util/losses.py","file_url":"https://github.com/fadedcosine/pos-guided-neural-text-generation/blob/HEAD/util/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"785d35781789b4f4"}},{"code_sha256_prefix":"1debde02b971ce55","entry":"UnlikelihoodLoss","repo":"c00k1ez/plain-transformers","repo_kind":"listed","path":"src/plain_transformers/losses/unlikelihood_loss.py","file_url":"https://github.com/c00k1ez/plain-transformers/blob/HEAD/src/plain_transformers/losses/unlikelihood_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1debde02b971ce55"}},{"code_sha256_prefix":"bc59957c30488434","entry":"compute_unlikelihood_loss","repo":"ljyflores/loss-library","repo_kind":"listed","path":"src/loss_library/utils_unlikelihood_loss.py","file_url":"https://github.com/ljyflores/loss-library/blob/HEAD/src/loss_library/utils_unlikelihood_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc59957c30488434"}},{"code_sha256_prefix":"a261ba0741e044d7","entry":"label_smoothed_nll_loss","repo":"griff4692/calibrating-summaries","repo_kind":"listed","path":"model/contrast_utils.py","file_url":"https://github.com/griff4692/calibrating-summaries/blob/HEAD/model/contrast_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a261ba0741e044d7"}},{"code_sha256_prefix":"315417bd4bc9b0ad","entry":"label_smoothed_unlikelihood","repo":"griff4692/calibrating-summaries","repo_kind":"listed","path":"model/contrast_utils.py","file_url":"https://github.com/griff4692/calibrating-summaries/blob/HEAD/model/contrast_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"315417bd4bc9b0ad"}},{"code_sha256_prefix":"ace3f464bf5b4c7a","entry":"top_k_logits","repo":"facebookresearch/unlikelihood_training","repo_kind":"official","path":"custom/sequence_generator.py","file_url":"https://github.com/facebookresearch/unlikelihood_training/blob/HEAD/custom/sequence_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ace3f464bf5b4c7a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}