{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lmrl-gym-benchmarks-for-multi-turn","title":"LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models","arxiv_id":"2311.18232","date":"2023-11-30","proceeding":null,"authors":["Marwa Abdulhai","Isadora White","Charlie Snell","Charles Sun","Joey Hong","Yuexiang Zhai","Kelvin Xu","Sergey Levine"],"abstract":"Large language models (LLMs) provide excellent text-generation capabilities, but standard prompting and generation methods generally do not lead to intentional or goal-directed agents and might necessitate considerable prompt tuning. This becomes particularly apparent in multi-turn conversations: even the best current LLMs rarely ask clarifying questions, engage in explicit information gathering, or take actions now that lead to better decisions after multiple turns. Reinforcement learning has the potential to leverage the powerful modeling capabilities of LLMs, as well as their internal representation of textual interactions, to create capable goal-directed language agents. This can enable intentional and temporally extended interactions, such as with humans, through coordinated persuasion and carefully crafted questions, or in goal-directed play through text games to bring about desired final outcomes. However, enabling this requires the community to develop stable and reliable reinforcement learning algorithms that can effectively train LLMs. Developing such algorithms requires tasks that can gauge progress on algorithm design, provide accessible and reproducible evaluations for multi-turn interactions, and cover a range of task properties and challenges in improving reinforcement learning algorithms. Our paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for LLMs, together with an open-source research framework containing a basic toolkit for getting started on multi-turn RL with offline value-based and policy-based RL methods. Our benchmark consists of 8 different language tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.","url_abs":"https://arxiv.org/abs/2311.18232v1","url_pdf":"https://arxiv.org/pdf/2311.18232v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lmrl-gym-benchmarks-for-multi-turn","repo_url":"https://github.com/abdulhaim/lmrl-gym","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.18232","atlas_url":"https://app.syntology.ai/?focus=2311.18232","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.18232"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abdulhaim/lmrl-gym","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a279fdf4e1e2f56e","entry":"filter_items","repo":"abdulhaim/lmrl-gym","repo_kind":"official","path":"LLM_RL/algorithms/bc/data.py","file_url":"https://github.com/abdulhaim/lmrl-gym/blob/HEAD/LLM_RL/algorithms/bc/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a279fdf4e1e2f56e"}},{"code_sha256_prefix":"c04433a04b87059b","entry":"get_query_indicators","repo":"abdulhaim/lmrl-gym","repo_kind":"official","path":"LLM_RL/algorithms/ilql/base_interface.py","file_url":"https://github.com/abdulhaim/lmrl-gym/blob/HEAD/LLM_RL/algorithms/ilql/base_interface.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c04433a04b87059b"}},{"code_sha256_prefix":"cb7f4af05dcf8ec4","entry":"get_tensor_stats","repo":"abdulhaim/lmrl-gym","repo_kind":"official","path":"LLM_RL/utils.py","file_url":"https://github.com/abdulhaim/lmrl-gym/blob/HEAD/LLM_RL/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb7f4af05dcf8ec4"}},{"code_sha256_prefix":"732e9b1128ba66a5","entry":"get_tensor_stats_np","repo":"abdulhaim/lmrl-gym","repo_kind":"official","path":"LLM_RL/utils.py","file_url":"https://github.com/abdulhaim/lmrl-gym/blob/HEAD/LLM_RL/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"732e9b1128ba66a5"}},{"code_sha256_prefix":"472b4b2ee09e79f6","entry":"label_logs","repo":"abdulhaim/lmrl-gym","repo_kind":"official","path":"LLM_RL/log_utils.py","file_url":"https://github.com/abdulhaim/lmrl-gym/blob/HEAD/LLM_RL/log_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"472b4b2ee09e79f6"}},{"code_sha256_prefix":"0010e3a7219eac47","entry":"pull_logs","repo":"abdulhaim/lmrl-gym","repo_kind":"official","path":"LLM_RL/log_utils.py","file_url":"https://github.com/abdulhaim/lmrl-gym/blob/HEAD/LLM_RL/log_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0010e3a7219eac47"}},{"code_sha256_prefix":"9e4eea7687d40ae5","entry":"unpad_array","repo":"abdulhaim/lmrl-gym","repo_kind":"official","path":"LLM_RL/utils.py","file_url":"https://github.com/abdulhaim/lmrl-gym/blob/HEAD/LLM_RL/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e4eea7687d40ae5"}},{"code_sha256_prefix":"a6acdf2ec8531626","entry":"combine_logs","repo":"abdulhaim/lmrl-gym","repo_kind":"official","path":"LLM_RL/log_utils.py","file_url":"https://github.com/abdulhaim/lmrl-gym/blob/HEAD/LLM_RL/log_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6acdf2ec8531626"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}