{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-off-policy-meta-reinforcement","title":"Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables","arxiv_id":"1903.08254","date":"2019-03-19","proceeding":"ICLR Workshop LLD 2019","authors":["Kate Rakelly","Aurick Zhou","Deirdre Quillen","Chelsea Finn","Sergey Levine"],"abstract":"Deep reinforcement learning algorithms require large amounts of experience to\nlearn an individual task. While in principle meta-reinforcement learning\n(meta-RL) algorithms enable agents to learn new skills from small amounts of\nexperience, several major challenges preclude their practicality. Current\nmethods rely heavily on on-policy experience, limiting their sample efficiency.\nThe also lack mechanisms to reason about task uncertainty when adapting to new\ntasks, limiting their effectiveness in sparse reward problems. In this paper,\nwe address these challenges by developing an off-policy meta-RL algorithm that\ndisentangles task inference and control. In our approach, we perform online\nprobabilistic filtering of latent task variables to infer how to solve a new\ntask from small amounts of experience. This probabilistic interpretation\nenables posterior sampling for structured and efficient exploration. We\ndemonstrate how to integrate these task variables with off-policy RL algorithms\nto achieve both meta-training and adaptation efficiency. Our method outperforms\nprior algorithms in sample efficiency by 20-100X as well as in asymptotic\nperformance on several meta-RL benchmarks.","url_abs":"http://arxiv.org/abs/1903.08254v1","url_pdf":"http://arxiv.org/pdf/1903.08254v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-off-policy-meta-reinforcement","repo_url":"https://github.com/katerakelly/oyster","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"efficient-off-policy-meta-reinforcement","repo_url":"https://github.com/Clyde21c/PEARL_bullet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"efficient-off-policy-meta-reinforcement","repo_url":"https://github.com/ido90/roml-pearl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"efficient-off-policy-meta-reinforcement","repo_url":"https://github.com/junmokane/offline-PEARL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"efficient-off-policy-meta-reinforcement","repo_url":"https://github.com/lujiayou123/Off-Policy-Meta-Reinforcement-Learning-via-Unsupervised-Domain-Translation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"efficient-off-policy-meta-reinforcement","repo_url":"https://github.com/victorchan314/cs287_final_project","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"efficient-off-policy-meta-reinforcement","repo_url":"https://github.com/waterhorse1/Pearl_relabel","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"efficient-exploration","task_name":"Efficient Exploration"},{"task_slug":"meta-reinforcement-learning","task_name":"Meta Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1903.08254","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1903.08254"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/katerakelly/oyster","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/victorchan314/cs287_final_project","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/waterhorse1/Pearl_relabel","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ido90/roml-pearl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Clyde21c/PEARL_bullet","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lujiayou123/Off-Policy-Meta-Reinforcement-Learning-via-Unsupervised-Domain-Translation","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/junmokane/offline-PEARL","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"listed":{"samples":9,"ran":6,"repositories":4}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"f395bfda2e4813d7","entry":"create_stats_ordered_dict","repo":"lujiayou123/Off-Policy-Meta-Reinforcement-Learning-via-Unsupervised-Domain-Translation","repo_kind":"listed","path":"rlkit/core/eval_util.py","file_url":"https://github.com/lujiayou123/Off-Policy-Meta-Reinforcement-Learning-via-Unsupervised-Domain-Translation/blob/HEAD/rlkit/core/eval_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f395bfda2e4813d7"}},{"code_sha256_prefix":"d8a2e44ea15298ac","entry":"cvar","repo":"ido90/roml-pearl","repo_kind":"listed","path":"rlkit/core/rl_algorithm.py","file_url":"https://github.com/ido90/roml-pearl/blob/HEAD/rlkit/core/rl_algorithm.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d8a2e44ea15298ac"}},{"code_sha256_prefix":"e937b70b07bfc9d1","entry":"deep_update_dict","repo":"junmokane/offline-PEARL","repo_kind":"listed","path":"launch_experiment.py","file_url":"https://github.com/junmokane/offline-PEARL/blob/HEAD/launch_experiment.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e937b70b07bfc9d1"}},{"code_sha256_prefix":"a5d10a0b704c8db7","entry":"get_average_returns","repo":"lujiayou123/Off-Policy-Meta-Reinforcement-Learning-via-Unsupervised-Domain-Translation","repo_kind":"listed","path":"rlkit/core/eval_util.py","file_url":"https://github.com/lujiayou123/Off-Policy-Meta-Reinforcement-Learning-via-Unsupervised-Domain-Translation/blob/HEAD/rlkit/core/eval_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5d10a0b704c8db7"}},{"code_sha256_prefix":"c809703793017cd2","entry":"get_generic_path_information","repo":"lujiayou123/Off-Policy-Meta-Reinforcement-Learning-via-Unsupervised-Domain-Translation","repo_kind":"listed","path":"rlkit/core/eval_util.py","file_url":"https://github.com/lujiayou123/Off-Policy-Meta-Reinforcement-Learning-via-Unsupervised-Domain-Translation/blob/HEAD/rlkit/core/eval_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c809703793017cd2"}},{"code_sha256_prefix":"5cb0c55f58279f59","entry":"simple_separated_format","repo":"victorchan314/cs287_final_project","repo_kind":"listed","path":"rlkit/core/tabulate.py","file_url":"https://github.com/victorchan314/cs287_final_project/blob/HEAD/rlkit/core/tabulate.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cb0c55f58279f59"}},{"code_sha256_prefix":"8a6359263de246f7","entry":"get_dim","repo":"junmokane/offline-PEARL","repo_kind":"listed","path":"rlkit/data_management/env_offline_replay_buffer.py","file_url":"https://github.com/junmokane/offline-PEARL/blob/HEAD/rlkit/data_management/env_offline_replay_buffer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a6359263de246f7"}},{"code_sha256_prefix":"3d87729bd4c217d5","entry":"get_mujoco_zip_name","repo":"junmokane/offline-PEARL","repo_kind":"listed","path":"docker/install_mujoco.py","file_url":"https://github.com/junmokane/offline-PEARL/blob/HEAD/docker/install_mujoco.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d87729bd4c217d5"}},{"code_sha256_prefix":"5480531658d82335","entry":"register_license","repo":"junmokane/offline-PEARL","repo_kind":"listed","path":"rand_param_envs/mujoco_py/mjcore.py","file_url":"https://github.com/junmokane/offline-PEARL/blob/HEAD/rand_param_envs/mujoco_py/mjcore.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5480531658d82335"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}