{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-contrastive-framework-for-neural-text","title":"A Contrastive Framework for Neural Text Generation","arxiv_id":"2202.06417","date":"2022-02-13","proceeding":null,"authors":["Yixuan Su","Tian Lan","Yan Wang","Dani Yogatama","Lingpeng Kong","Nigel Collier"],"abstract":"Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the-art text generation methods as evaluated by both human and automatic metrics.","url_abs":"https://arxiv.org/abs/2202.06417v3","url_pdf":"https://arxiv.org/pdf/2202.06417v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-contrastive-framework-for-neural-text","repo_url":"https://github.com/yxuansu/simctg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"a-contrastive-framework-for-neural-text","repo_url":"https://github.com/yxuansu/magic","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.06417","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.06417"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yxuansu/magic","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yxuansu/simctg","reach":null}],"summary":{"ran_draft_wrong":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"b961e44bf1174eed","entry":"build_mask_matrix","repo":"yxuansu/magic","repo_kind":"listed","path":"image_captioning/language_model/simctg.py","file_url":"https://github.com/yxuansu/magic/blob/HEAD/image_captioning/language_model/simctg.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b961e44bf1174eed"}},{"code_sha256_prefix":"369dc33768fbe11b","entry":"contrastive_loss","repo":"yxuansu/magic","repo_kind":"listed","path":"image_captioning/language_model/simctg.py","file_url":"https://github.com/yxuansu/magic/blob/HEAD/image_captioning/language_model/simctg.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"369dc33768fbe11b"}},{"code_sha256_prefix":"fd90cea1b58123b4","entry":"SimCTG","repo":"yxuansu/magic","repo_kind":"listed","path":"image_captioning/language_model/simctg.py","file_url":"https://github.com/yxuansu/magic/blob/HEAD/image_captioning/language_model/simctg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd90cea1b58123b4"}},{"code_sha256_prefix":"e597868504efc9e6","entry":"SimCTGGPT","repo":"yxuansu/simctg","repo_kind":"official","path":"simctg/simctggpt.py","file_url":"https://github.com/yxuansu/simctg/blob/HEAD/simctg/simctggpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e597868504efc9e6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}