{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ganlm-encoder-decoder-pre-training-with-an","title":"GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator","arxiv_id":"2212.10218","date":"2022-12-20","proceeding":null,"authors":["Jian Yang","Shuming Ma","Li Dong","Shaohan Huang","Haoyang Huang","Yuwei Yin","Dongdong Zhang","Liqun Yang","Furu Wei","Zhoujun Li"],"abstract":"Pre-trained models have achieved remarkable success in natural language processing (NLP). However, existing pre-training methods underutilize the benefits of language understanding for generation. Inspired by the idea of Generative Adversarial Networks (GANs), we propose a GAN-style model for encoder-decoder pre-training by introducing an auxiliary discriminator, unifying the ability of language understanding and generation in a single model. Our model, named as GanLM, is trained with two pre-training objectives: replaced token detection and replaced token denoising. Specifically, given masked source sentences, the generator outputs the target distribution and the discriminator predicts whether the target sampled tokens from distribution are incorrect. The target sentence is replaced with misclassified tokens to construct noisy previous context, which is used to generate the gold sentence. In general, both tasks improve the ability of language understanding and generation by selectively using the denoising data. Extensive experiments in language generation benchmarks show that GanLM with the powerful language understanding capability outperforms various strong pre-trained language models (PLMs) and achieves state-of-the-art performance.","url_abs":"https://arxiv.org/abs/2212.10218v2","url_pdf":"https://arxiv.org/pdf/2212.10218v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ganlm-encoder-decoder-pre-training-with-an","repo_url":"https://github.com/csjianyang/ganlm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.10218","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.10218"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/csjianyang/ganlm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bedeff473c33061f","entry":"clean","repo":"csjianyang/ganlm","repo_kind":"official","path":"evaluation/bs_pyrouge.py","file_url":"https://github.com/csjianyang/ganlm/blob/HEAD/evaluation/bs_pyrouge.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bedeff473c33061f"}},{"code_sha256_prefix":"7d0f0d30d5c3027f","entry":"count_tokens","repo":"csjianyang/ganlm","repo_kind":"official","path":"evaluation/cnn_dm.py","file_url":"https://github.com/csjianyang/ganlm/blob/HEAD/evaluation/cnn_dm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d0f0d30d5c3027f"}},{"code_sha256_prefix":"c19b75e22d2023fb","entry":"rouge_results_to_str","repo":"csjianyang/ganlm","repo_kind":"official","path":"evaluation/cnn_dm.py","file_url":"https://github.com/csjianyang/ganlm/blob/HEAD/evaluation/cnn_dm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c19b75e22d2023fb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}