{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/grounding-large-language-models-in","title":"Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning","arxiv_id":"2302.02662","date":"2023-02-06","proceeding":null,"authors":["Thomas Carta","Clément Romac","Thomas Wolf","Sylvain Lamprier","Olivier Sigaud","Pierre-Yves Oudeyer"],"abstract":"Recent works successfully leveraged Large Language Models' (LLM) abilities to capture abstract knowledge about world's physics to solve decision-making problems. Yet, the alignment between LLMs' knowledge and the environment can be wrong and limit functional competence due to lack of grounding. In this paper, we study an approach (named GLAM) to achieve this alignment through functional grounding: we consider an agent using an LLM as a policy that is progressively updated as the agent interacts with the environment, leveraging online Reinforcement Learning to improve its performance to solve goals. Using an interactive textual environment designed to study higher-level forms of functional grounding, and a set of spatial and navigation tasks, we study several scientific questions: 1) Can LLMs boost sample efficiency for online learning of various RL tasks? 2) How can it boost different forms of generalization? 3) What is the impact of online learning? We study these questions by functionally grounding several variants (size, architecture) of FLAN-T5.","url_abs":"https://arxiv.org/abs/2302.02662v4","url_pdf":"https://arxiv.org/pdf/2302.02662v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"grounding-large-language-models-in","repo_url":"https://github.com/clementromac/lamorel","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"grounding-large-language-models-in","repo_url":"https://github.com/flowersteam/grounding_llms_with_online_rl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"grounding-large-language-models-in","repo_url":"https://github.com/flowersteam/lamorel","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.02662","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.02662"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/flowersteam/grounding_llms_with_online_rl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/flowersteam/lamorel","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/clementromac/lamorel","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":2,"ran_fixture":1,"ran_honours":1,"unverified":5},"by_repo_kind":{"official":{"samples":12,"ran":7,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b087b0213c49d353","entry":"BaseUpdater","repo":"flowersteam/lamorel","repo_kind":"official","path":"lamorel/src/lamorel/server/llms/updaters/base_updater.py","file_url":"https://github.com/flowersteam/lamorel/blob/HEAD/lamorel/src/lamorel/server/llms/updaters/base_updater.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b087b0213c49d353"}},{"code_sha256_prefix":"f35eacbd3758dba2","entry":"DictList","repo":"flowersteam/grounding_llms_with_online_rl","repo_kind":"official","path":"experiments/agents/ppo/llm_ppo_agent.py","file_url":"https://github.com/flowersteam/grounding_llms_with_online_rl/blob/HEAD/experiments/agents/ppo/llm_ppo_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f35eacbd3758dba2"}},{"code_sha256_prefix":"f93854005e6f5869","entry":"ExtraInfoCollector","repo":"flowersteam/grounding_llms_with_online_rl","repo_kind":"official","path":"experiments/agents/ppo/llm_ppo_agent.py","file_url":"https://github.com/flowersteam/grounding_llms_with_online_rl/blob/HEAD/experiments/agents/ppo/llm_ppo_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f93854005e6f5869"}},{"code_sha256_prefix":"efa2c567105100f0","entry":"emph","repo":"flowersteam/grounding_llms_with_online_rl","repo_kind":"official","path":"experiments/agents/ppo/llm_ppo_agent.py","file_url":"https://github.com/flowersteam/grounding_llms_with_online_rl/blob/HEAD/experiments/agents/ppo/llm_ppo_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efa2c567105100f0"}},{"code_sha256_prefix":"a01836a09eb71d45","entry":"info","repo":"flowersteam/grounding_llms_with_online_rl","repo_kind":"official","path":"experiments/agents/ppo/llm_ppo_agent.py","file_url":"https://github.com/flowersteam/grounding_llms_with_online_rl/blob/HEAD/experiments/agents/ppo/llm_ppo_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a01836a09eb71d45"}},{"code_sha256_prefix":"351f0073ef3dccf5","entry":"reward_function","repo":"flowersteam/grounding_llms_with_online_rl","repo_kind":"official","path":"experiments/train_language_agent.py","file_url":"https://github.com/flowersteam/grounding_llms_with_online_rl/blob/HEAD/experiments/train_language_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"351f0073ef3dccf5"}},{"code_sha256_prefix":"fdf8fa20114f1fbe","entry":"reward_function_shapped","repo":"flowersteam/grounding_llms_with_online_rl","repo_kind":"official","path":"experiments/train_language_agent.py","file_url":"https://github.com/flowersteam/grounding_llms_with_online_rl/blob/HEAD/experiments/train_language_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fdf8fa20114f1fbe"}},{"code_sha256_prefix":"3a32c056e4db9777","entry":"BaseAgent","repo":"flowersteam/grounding_llms_with_online_rl","repo_kind":"official","path":"experiments/agents/ppo/llm_ppo_agent.py","file_url":"https://github.com/flowersteam/grounding_llms_with_online_rl/blob/HEAD/experiments/agents/ppo/llm_ppo_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a32c056e4db9777"}},{"code_sha256_prefix":"e7ee48ded5e587f1","entry":"BasePPOAgent","repo":"flowersteam/grounding_llms_with_online_rl","repo_kind":"official","path":"experiments/agents/ppo/llm_ppo_agent.py","file_url":"https://github.com/flowersteam/grounding_llms_with_online_rl/blob/HEAD/experiments/agents/ppo/llm_ppo_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7ee48ded5e587f1"}},{"code_sha256_prefix":"7c1ab31fe12a53a6","entry":"LLMPPOAgent","repo":"flowersteam/grounding_llms_with_online_rl","repo_kind":"official","path":"experiments/agents/ppo/llm_ppo_agent.py","file_url":"https://github.com/flowersteam/grounding_llms_with_online_rl/blob/HEAD/experiments/agents/ppo/llm_ppo_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c1ab31fe12a53a6"}},{"code_sha256_prefix":"a71506f048bfc377","entry":"load_hf_model_and_tokenizer","repo":"clementromac/lamorel","repo_kind":"official","path":"lamorel/src/lamorel/server/llms/utils/load_hf_model_and_tokenizer.py","file_url":"https://github.com/clementromac/lamorel/blob/HEAD/lamorel/src/lamorel/server/llms/utils/load_hf_model_and_tokenizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a71506f048bfc377"}},{"code_sha256_prefix":"5451850f7e5ef008","entry":"prepare_args_for_parsing","repo":"clementromac/lamorel","repo_kind":"official","path":"lamorel/src/lamorel_launcher/launch.py","file_url":"https://github.com/clementromac/lamorel/blob/HEAD/lamorel/src/lamorel_launcher/launch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5451850f7e5ef008"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}