{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/text-generation-by-learning-from-off-policy","title":"Text Generation by Learning from Demonstrations","arxiv_id":"2009.07839","date":"2020-09-16","proceeding":"ICLR 2021 1","authors":["Richard Yuanzhe Pang","He He"],"abstract":"Current approaches to text generation largely rely on autoregressive models and maximum likelihood estimation. This paradigm leads to (i) diverse but low-quality samples due to mismatched learning objective and evaluation metric (likelihood vs. quality) and (ii) exposure bias due to mismatched history distributions (gold vs. model-generated). To alleviate these problems, we frame text generation as an offline reinforcement learning (RL) problem with expert demonstrations (i.e., the reference), where the goal is to maximize quality given model-generated histories. We propose GOLD (generation by off-policy learning from demonstrations): an easy-to-optimize algorithm that learns from the demonstrations by importance weighting. Intuitively, GOLD upweights confident tokens and downweights unconfident ones in the reference during training, avoiding optimization issues faced by prior RL approaches that rely on online data collection. According to both automatic and human evaluation, models trained by GOLD outperform those trained by MLE and policy gradient on summarization, question generation, and machine translation. Further, our models are less sensitive to decoding algorithms and alleviate exposure bias.","url_abs":"https://arxiv.org/abs/2009.07839v2","url_pdf":"https://arxiv.org/pdf/2009.07839v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"text-generation-by-learning-from-off-policy","repo_url":"https://github.com/yzpang/gold-off-policy-text-gen-iclr21","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"question-generation","task_name":"Question Generation"},{"task_slug":"question-generation","task_name":"Question-Generation"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2009.07839","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2009.07839"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/clovaai/FocusSeq2Seq","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/pytorch/fairseq","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yzpang/gold-off-policy-text-gen-iclr21","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5b9354b55563c25f","entry":"GRUEncoder","repo":"clovaai/FocusSeq2Seq","repo_kind":"found_in_text","path":"models.py","file_url":"https://github.com/clovaai/FocusSeq2Seq/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b9354b55563c25f"}},{"code_sha256_prefix":"8c86add14c1d23db","entry":"ParallelDecoder","repo":"clovaai/FocusSeq2Seq","repo_kind":"found_in_text","path":"models.py","file_url":"https://github.com/clovaai/FocusSeq2Seq/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c86add14c1d23db"}},{"code_sha256_prefix":"fbd0b841504c9ce7","entry":"ParallelSelector","repo":"clovaai/FocusSeq2Seq","repo_kind":"found_in_text","path":"models.py","file_url":"https://github.com/clovaai/FocusSeq2Seq/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbd0b841504c9ce7"}},{"code_sha256_prefix":"f0d5ad048814d336","entry":"repeat","repo":"clovaai/FocusSeq2Seq","repo_kind":"found_in_text","path":"models.py","file_url":"https://github.com/clovaai/FocusSeq2Seq/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0d5ad048814d336"}},{"code_sha256_prefix":"710c6122db670130","entry":"FocusSelector","repo":"clovaai/FocusSeq2Seq","repo_kind":"found_in_text","path":"models.py","file_url":"https://github.com/clovaai/FocusSeq2Seq/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"710c6122db670130"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}