{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/online-intrinsic-rewards-for-decision-making","title":"Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback","arxiv_id":"2410.23022","date":"2024-10-30","proceeding":null,"authors":["Qinqing Zheng","Mikael Henaff","Amy Zhang","Aditya Grover","Brandon Amos"],"abstract":"Automatically synthesizing dense rewards from natural language descriptions is a promising paradigm in reinforcement learning (RL), with applications to sparse reward problems, open-ended exploration, and hierarchical skill design. Recent works have made promising steps by exploiting the prior knowledge of large language models (LLMs). However, these approaches suffer from important limitations: they are either not scalable to problems requiring billions of environment samples, due to requiring LLM annotations for each observation, or they require a diverse offline dataset, which may not exist or be impossible to collect. In this work, we address these limitations through a combination of algorithmic and systems-level contributions. We propose \\oni, a distributed architecture that simultaneously learns an RL policy and an intrinsic reward function using LLM feedback. Our approach annotates the agent's collected experience via an asynchronous LLM server, which is then distilled into an intrinsic reward model. We explore a range of algorithmic choices for reward modeling with varying complexity, including hashing, classification, and ranking models. By studying their relative tradeoffs, we shed light on questions regarding intrinsic reward design for sparse reward problems. Our approach achieves state-of-the-art performance across a range of challenging, sparse reward tasks from the NetHack Learning Environment in a simple unified process, solely using the agent's gathered experience, without requiring external datasets. We make our code available at \\url{https://github.com/facebookresearch/oni}.","url_abs":"https://arxiv.org/abs/2410.23022v2","url_pdf":"https://arxiv.org/pdf/2410.23022v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"online-intrinsic-rewards-for-decision-making","repo_url":"https://github.com/facebookresearch/oni","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"nethack","task_name":"NetHack"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.23022","atlas_url":"https://app.syntology.ai/?focus=2410.23022","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.23022"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/oni","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"7733077acd913a28","entry":"calc_conv_output_size","repo":"facebookresearch/oni","repo_kind":"official","path":"rl_baseline/encoders_nle.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/rl_baseline/encoders_nle.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7733077acd913a28"}},{"code_sha256_prefix":"20065027a18e8242","entry":"decode_message","repo":"facebookresearch/oni","repo_kind":"official","path":"utils/decode.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/utils/decode.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"20065027a18e8242"}},{"code_sha256_prefix":"bc2433f1ec8aa864","entry":"dict_collate_fn","repo":"facebookresearch/oni","repo_kind":"official","path":"rlaif/dataset.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/rlaif/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bc2433f1ec8aa864"}},{"code_sha256_prefix":"9eee67ebcca5449c","entry":"encode_message_batch","repo":"facebookresearch/oni","repo_kind":"official","path":"utils/decode.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/utils/decode.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9eee67ebcca5449c"}},{"code_sha256_prefix":"7c381ba18918f4ba","entry":"encode_message_no_pad","repo":"facebookresearch/oni","repo_kind":"official","path":"utils/decode.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/utils/decode.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7c381ba18918f4ba"}},{"code_sha256_prefix":"9ac543dc609dd221","entry":"flatten_pair_collate_fn","repo":"facebookresearch/oni","repo_kind":"official","path":"rlaif/dataset.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/rlaif/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9ac543dc609dd221"}},{"code_sha256_prefix":"5ed388d816ba33a9","entry":"generate_prompt_classification","repo":"facebookresearch/oni","repo_kind":"official","path":"rlaif/prompts.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/rlaif/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5ed388d816ba33a9"}},{"code_sha256_prefix":"83d7f2ee7157190c","entry":"generate_prompt_ranking","repo":"facebookresearch/oni","repo_kind":"official","path":"rlaif/prompts.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/rlaif/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"83d7f2ee7157190c"}},{"code_sha256_prefix":"328a80c046fb66a5","entry":"get_ranking_goal_strings","repo":"facebookresearch/oni","repo_kind":"official","path":"rlaif/prompts.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/rlaif/prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"328a80c046fb66a5"}},{"code_sha256_prefix":"7391a50aefeb8f4b","entry":"initialize_char_array","repo":"facebookresearch/oni","repo_kind":"official","path":"rl_baseline/obs_wrappers.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/rl_baseline/obs_wrappers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7391a50aefeb8f4b"}},{"code_sha256_prefix":"286f16b8cdd48f8f","entry":"mlp","repo":"facebookresearch/oni","repo_kind":"official","path":"rlaif/reward_model.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/rlaif/reward_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"286f16b8cdd48f8f"}},{"code_sha256_prefix":"d4ba15725b44064f","entry":"tuple_dict_collate_fn","repo":"facebookresearch/oni","repo_kind":"official","path":"rlaif/dataset.py","file_url":"https://github.com/facebookresearch/oni/blob/HEAD/rlaif/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d4ba15725b44064f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}