{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-break-the-loop-analyzing-and","title":"Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation","arxiv_id":"2206.02369","date":"2022-06-06","proceeding":null,"authors":["Jin Xu","Xiaojiang Liu","Jianhao Yan","Deng Cai","Huayang Li","Jian Li"],"abstract":"While large-scale neural language models, such as GPT2 and BART, have achieved impressive results on various text generation tasks, they tend to get stuck in undesirable sentence-level loops with maximization-based decoding algorithms (\\textit{e.g.}, greedy search). This phenomenon is counter-intuitive since there are few consecutive sentence-level repetitions in human corpora (e.g., 0.02\\% in Wikitext-103). To investigate the underlying reasons for generating consecutive sentence-level repetitions, we study the relationship between the probabilities of the repetitive tokens and their previous repetitions in the context. Through our quantitative experiments, we find that 1) Language models have a preference to repeat the previous sentence; 2) The sentence-level repetitions have a \\textit{self-reinforcement effect}: the more times a sentence is repeated in the context, the higher the probability of continuing to generate that sentence; 3) The sentences with higher initial probabilities usually have a stronger self-reinforcement effect. Motivated by our findings, we propose a simple and effective training method \\textbf{DITTO} (Pseu\\underline{D}o-Repet\\underline{IT}ion Penaliza\\underline{T}i\\underline{O}n), where the model learns to penalize probabilities of sentence-level repetitions from pseudo repetitive data. Although our method is motivated by mitigating repetitions, experiments show that DITTO not only mitigates the repetition issue without sacrificing perplexity, but also achieves better generation quality. Extensive experiments on open-ended text generation (Wikitext-103) and text summarization (CNN/DailyMail) demonstrate the generality and effectiveness of our method.","url_abs":"https://arxiv.org/abs/2206.02369v2","url_pdf":"https://arxiv.org/pdf/2206.02369v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-break-the-loop-analyzing-and","repo_url":"https://github.com/Jxu-Thu/DITTO","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-to-break-the-loop-analyzing-and","repo_url":"https://github.com/Jxu-Thu/DIITTO","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-summarization","task_name":"Text Summarization"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.02369","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.02369"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Jxu-Thu/DITTO","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Jxu-Thu/DIITTO","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":6},"by_repo_kind":{"listed":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"82bdf940665aeeb4","entry":"safe_readline","repo":"Jxu-Thu/DIITTO","repo_kind":"listed","path":"fairseq/binarizer.py","file_url":"https://github.com/Jxu-Thu/DIITTO/blob/HEAD/fairseq/binarizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"82bdf940665aeeb4"}},{"code_sha256_prefix":"9eda7f80331d658c","entry":"dataset_dest_prefix","repo":"Jxu-Thu/DIITTO","repo_kind":"listed","path":"preprocess.py","file_url":"https://github.com/Jxu-Thu/DIITTO/blob/HEAD/preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"9eda7f80331d658c"}},{"code_sha256_prefix":"c5ea9d48b6fc2167","entry":"load_checkpoint","repo":"Jxu-Thu/DIITTO","repo_kind":"listed","path":"fairseq/checkpoint_utils.py","file_url":"https://github.com/Jxu-Thu/DIITTO/blob/HEAD/fairseq/checkpoint_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"c5ea9d48b6fc2167"}},{"code_sha256_prefix":"df40d96cc20e7f57","entry":"load_checkpoint_to_cpu","repo":"Jxu-Thu/DIITTO","repo_kind":"listed","path":"fairseq/checkpoint_utils.py","file_url":"https://github.com/Jxu-Thu/DIITTO/blob/HEAD/fairseq/checkpoint_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"df40d96cc20e7f57"}},{"code_sha256_prefix":"b3616a62387703cb","entry":"save_checkpoint","repo":"Jxu-Thu/DIITTO","repo_kind":"listed","path":"fairseq/checkpoint_utils.py","file_url":"https://github.com/Jxu-Thu/DIITTO/blob/HEAD/fairseq/checkpoint_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"b3616a62387703cb"}},{"code_sha256_prefix":"c0e67067471200e3","entry":"sentence_repetition_count","repo":"Jxu-Thu/DIITTO","repo_kind":"listed","path":"report_metrics.py","file_url":"https://github.com/Jxu-Thu/DIITTO/blob/HEAD/report_metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"c0e67067471200e3"}},{"code_sha256_prefix":"38878cf98d083511","entry":"sentence_repetition_count_all","repo":"Jxu-Thu/DIITTO","repo_kind":"listed","path":"report_metrics.py","file_url":"https://github.com/Jxu-Thu/DIITTO/blob/HEAD/report_metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"38878cf98d083511"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}