{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/esrl-efficient-sampling-based-reinforcement","title":"ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generation","arxiv_id":"2308.02223","date":"2023-08-04","proceeding":null,"authors":["Chenglong Wang","Hang Zhou","Yimin Hu","Yifu Huo","Bei Li","Tongran Liu","Tong Xiao","Jingbo Zhu"],"abstract":"Applying Reinforcement Learning (RL) to sequence generation models enables the direct optimization of long-term rewards (\\textit{e.g.,} BLEU and human feedback), but typically requires large-scale sampling over a space of action sequences. This is a computational challenge as presented by the practice of sequence generation problems, such as machine translation, where we often deal with a large action space (\\textit{e.g.,} a vocabulary) and a long action sequence (\\textit{e.g.,} a translation). In this work, we introduce two-stage sampling and dynamic sampling approaches to improve the sampling efficiency during training sequence generation models via RL. We experiment with our approaches on the traditional sequence generation tasks, including machine translation and abstractive summarization. Furthermore, we evaluate our approaches in RL from human feedback (RLHF) through training a large language model using the reward model. Experimental results show that the efficient sampling-based RL, referred to as ESRL, can outperform all baselines in terms of both training efficiency and memory consumption. Notably, ESRL yields consistent performance gains over the strong REINFORCE, minimum risk training, and proximal policy optimization methods.","url_abs":"https://arxiv.org/abs/2308.02223v1","url_pdf":"https://arxiv.org/pdf/2308.02223v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"esrl-efficient-sampling-based-reinforcement","repo_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/tree/main/examples/esrl","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"esrl-efficient-sampling-based-reinforcement","repo_url":"https://github.com/wangclnlp/deepspeed-chat-extension","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"abstractive-text-summarization","task_name":"Abstractive Text Summarization"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"reinforce","method_name":"REINFORCE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.02223","atlas_url":"https://app.syntology.ai/?focus=2308.02223","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.02223"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/tree/main/examples/esrl","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wangclnlp/deepspeed-chat-extension","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran":10,"unverified":3},"by_repo_kind":{"official":{"samples":14,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2f0a690c70d11c23","entry":"gather_log_probs","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/training/step3_rlhf_finetuning/ppo_trainer.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/training/step3_rlhf_finetuning/ppo_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f0a690c70d11c23"}},{"code_sha256_prefix":"451c619b9609c3c0","entry":"generate","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/training/step1_supervised_finetuning/prompt_eval.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/training/step1_supervised_finetuning/prompt_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"451c619b9609c3c0"}},{"code_sha256_prefix":"3ba2013469906a98","entry":"generate","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/training/step1_supervised_finetuning/predict.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/training/step1_supervised_finetuning/predict.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ba2013469906a98"}},{"code_sha256_prefix":"385922750eb70702","entry":"generate_constrastive_search","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/training/step1_supervised_finetuning/prompt_eval.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/training/step1_supervised_finetuning/prompt_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"385922750eb70702"}},{"code_sha256_prefix":"8ce22956ed0b989e","entry":"get_eval_ds_config","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/training/utils/ds_utils.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/training/utils/ds_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ce22956ed0b989e"}},{"code_sha256_prefix":"2e5ee103f1579536","entry":"get_model_response","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/inference/chatbot.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/inference/chatbot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e5ee103f1579536"}},{"code_sha256_prefix":"0d4552e23174dc98","entry":"get_train_ds_config","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/training/utils/ds_utils.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/training/utils/ds_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d4552e23174dc98"}},{"code_sha256_prefix":"cdbaad9b00ba1398","entry":"now","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/training/utils/datetime_utils.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/training/utils/datetime_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdbaad9b00ba1398"}},{"code_sha256_prefix":"034cd7bbde2c40ff","entry":"prepare_datapair","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/training/step2_reward_model_finetuning/rw_eval.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/training/step2_reward_model_finetuning/rw_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"034cd7bbde2c40ff"}},{"code_sha256_prefix":"11e574c0b1fbdefc","entry":"prepare_singlesample","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/training/step2_reward_model_finetuning/rw_eval.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/training/step2_reward_model_finetuning/rw_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11e574c0b1fbdefc"}},{"code_sha256_prefix":"fa3a56183c58c15d","entry":"whiten","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/training/step3_rlhf_finetuning/ppo_trainer.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/training/step3_rlhf_finetuning/ppo_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa3a56183c58c15d"}},{"code_sha256_prefix":"01cbaddf368c1af0","entry":"get_generator","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/inference/chatbot.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/inference/chatbot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"01cbaddf368c1af0"}},{"code_sha256_prefix":"5a2aa8ea6f00f07c","entry":"get_user_input","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/inference/chatbot.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/inference/chatbot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a2aa8ea6f00f07c"}},{"code_sha256_prefix":"429d42a3bf925da5","entry":"log_init","repo":"wangclnlp/DeepSpeed-Chat-Extension","repo_kind":"official","path":"rlhf_llama/deepspeed_chat/training/step3_rlhf_finetuning/rlhf_engine.py","file_url":"https://github.com/wangclnlp/DeepSpeed-Chat-Extension/blob/HEAD/rlhf_llama/deepspeed_chat/training/step3_rlhf_finetuning/rlhf_engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"429d42a3bf925da5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}