{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-reinforcement-learning-for-sequence-to","title":"Deep Reinforcement Learning For Sequence to Sequence Models","arxiv_id":"1805.09461","date":"2018-05-24","proceeding":null,"authors":["Yaser Keneshloo","Tian Shi","Naren Ramakrishnan","Chandan K. Reddy"],"abstract":"In recent times, sequence-to-sequence (seq2seq) models have gained a lot of\npopularity and provide state-of-the-art performance in a wide variety of tasks\nsuch as machine translation, headline generation, text summarization, speech to\ntext conversion, and image caption generation. The underlying framework for all\nthese models is usually a deep neural network comprising an encoder and a\ndecoder. Although simple encoder-decoder models produce competitive results,\nmany researchers have proposed additional improvements over these\nsequence-to-sequence models, e.g., using an attention-based model over the\ninput, pointer-generation models, and self-attention models. However, such\nseq2seq models suffer from two common problems: 1) exposure bias and 2)\ninconsistency between train/test measurement. Recently, a completely novel\npoint of view has emerged in addressing these two problems in seq2seq models,\nleveraging methods from reinforcement learning (RL). In this survey, we\nconsider seq2seq problems from the RL point of view and provide a formulation\ncombining the power of RL methods in decision-making with sequence-to-sequence\nmodels that enable remembering long-term memories. We present some of the most\nrecent frameworks that combine concepts from RL and deep neural networks and\nexplain how these two areas could benefit from each other in solving complex\nseq2seq tasks. Our work aims to provide insights into some of the problems that\ninherently arise with current approaches and how we can address them with\nbetter RL models. We also provide the source code for implementing most of the\nRL models discussed in this paper to support the complex task of abstractive\ntext summarization.","url_abs":"http://arxiv.org/abs/1805.09461v4","url_pdf":"http://arxiv.org/pdf/1805.09461v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-reinforcement-learning-for-sequence-to","repo_url":"https://github.com/yaserkl/RLSeq2Seq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"deep-reinforcement-learning-for-sequence-to","repo_url":"https://github.com/IlyaGusev/summarus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"deep-reinforcement-learning-for-sequence-to","repo_url":"https://github.com/theamrzaki/text_summurization_abstractive_methods","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"abstractive-text-summarization","task_name":"Abstractive Text Summarization"},{"task_slug":"caption-generation","task_name":"Caption Generation"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"headline-generation","task_name":"Headline Generation"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"speech-to-text","task_name":"Speech-to-Text"},{"task_slug":"text-summarization","task_name":"Text Summarization"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"seq2seq","method_name":"Seq2Seq"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1805.09461","atlas_url":"https://app.syntology.ai/?focus=1805.09461","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1805.09461"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/IlyaGusev/summarus","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/theamrzaki/text_summurization_abstractive_methods","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yaserkl/RLSeq2Seq","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":2,"unverified":14},"by_repo_kind":{"official":{"samples":13,"ran":3,"repositories":1},"listed":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"84fcb74e05d73aff","entry":"make_html_safe","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/decode.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/decode.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84fcb74e05d73aff"}},{"code_sha256_prefix":"815c9f02431395eb","entry":"rouge_l_sentence_level","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/rouge.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/rouge.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"815c9f02431395eb"}},{"code_sha256_prefix":"eb37c01f4067f842","entry":"rouge_n","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/rouge.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/rouge.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb37c01f4067f842"}},{"code_sha256_prefix":"fe1c05743cd67af2","entry":"abstract2ids","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe1c05743cd67af2"}},{"code_sha256_prefix":"ecb10763ebbfe0bd","entry":"article2ids","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ecb10763ebbfe0bd"}},{"code_sha256_prefix":"67be1a4ec1456244","entry":"find_acs","repo":"IlyaGusev/summarus","repo_kind":"listed","path":"summarus/util/extraction_score.py","file_url":"https://github.com/IlyaGusev/summarus/blob/HEAD/summarus/util/extraction_score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67be1a4ec1456244"}},{"code_sha256_prefix":"d99eeb1146fb3504","entry":"fix_missing_period","repo":"IlyaGusev/summarus","repo_kind":"listed","path":"summarus/readers/cnn_dailymail_reader.py","file_url":"https://github.com/IlyaGusev/summarus/blob/HEAD/summarus/readers/cnn_dailymail_reader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d99eeb1146fb3504"}},{"code_sha256_prefix":"99ff57fcf802c668","entry":"get_article_and_abstract","repo":"IlyaGusev/summarus","repo_kind":"listed","path":"summarus/readers/cnn_dailymail_reader.py","file_url":"https://github.com/IlyaGusev/summarus/blob/HEAD/summarus/readers/cnn_dailymail_reader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"99ff57fcf802c668"}},{"code_sha256_prefix":"7d38d9759f7037fa","entry":"hashhex","repo":"IlyaGusev/summarus","repo_kind":"listed","path":"summarus/readers/cnn_dailymail_reader.py","file_url":"https://github.com/IlyaGusev/summarus/blob/HEAD/summarus/readers/cnn_dailymail_reader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d38d9759f7037fa"}},{"code_sha256_prefix":"eca0776bc82826da","entry":"outputids2words","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eca0776bc82826da"}},{"code_sha256_prefix":"5c25a03c38e34377","entry":"rouge_l_fscore","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/rouge_tensor.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/rouge_tensor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c25a03c38e34377"}},{"code_sha256_prefix":"77afbbd9f4048865","entry":"rouge_l_sentence_level","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/rouge_tensor.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/rouge_tensor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77afbbd9f4048865"}},{"code_sha256_prefix":"86e27006cb76b7f8","entry":"rouge_l_summary_level","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/rouge.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/rouge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"86e27006cb76b7f8"}},{"code_sha256_prefix":"eb5c66f55e72516a","entry":"rouge_n","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/rouge_tensor.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/rouge_tensor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb5c66f55e72516a"}},{"code_sha256_prefix":"85555d564514df13","entry":"scheduled_sampling","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/attention_decoder.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/attention_decoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85555d564514df13"}},{"code_sha256_prefix":"ef954b98b08bfa56","entry":"scheduled_sampling_vocab_dist","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/attention_decoder.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/attention_decoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef954b98b08bfa56"}},{"code_sha256_prefix":"8891bb52fbe37510","entry":"sort_hyps","repo":"yaserkl/RLSeq2Seq","repo_kind":"official","path":"src/beam_search.py","file_url":"https://github.com/yaserkl/RLSeq2Seq/blob/HEAD/src/beam_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8891bb52fbe37510"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}