{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/offline-meta-reinforcement-learning-with","title":"Offline Meta-Reinforcement Learning with Advantage Weighting","arxiv_id":"2008.06043","date":"2020-08-13","proceeding":null,"authors":["Eric Mitchell","Rafael Rafailov","Xue Bin Peng","Sergey Levine","Chelsea Finn"],"abstract":"This paper introduces the offline meta-reinforcement learning (offline meta-RL) problem setting and proposes an algorithm that performs well in this setting. Offline meta-RL is analogous to the widely successful supervised learning strategy of pre-training a model on a large batch of fixed, pre-collected data (possibly from various tasks) and fine-tuning the model to a new task with relatively little data. That is, in offline meta-RL, we meta-train on fixed, pre-collected data from several tasks in order to adapt to a new task with a very small amount (less than 5 trajectories) of data from the new task. By nature of being offline, algorithms for offline meta-RL can utilize the largest possible pool of training data available and eliminate potentially unsafe or costly data collection during meta-training. This setting inherits the challenges of offline RL, but it differs significantly because offline RL does not generally consider a) transfer to new tasks or b) limited data from the test task, both of which we face in offline meta-RL. Targeting the offline meta-RL setting, we propose Meta-Actor Critic with Advantage Weighting (MACAW), an optimization-based meta-learning algorithm that uses simple, supervised regression objectives for both the inner and outer loop of meta-training. On offline variants of common meta-RL benchmarks, we empirically find that this approach enables fully offline meta-reinforcement learning and achieves notable gains over prior methods.","url_abs":"https://arxiv.org/abs/2008.06043v3","url_pdf":"https://arxiv.org/pdf/2008.06043v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"offline-meta-reinforcement-learning-with","repo_url":"https://github.com/eric-mitchell/macaw","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"offline-meta-reinforcement-learning-with","repo_url":"https://github.com/eric-mitchell/macaw-min","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"meta-reinforcement-learning","task_name":"Meta Reinforcement Learning"},{"task_slug":"meta-learning","task_name":"Meta-Learning"},{"task_slug":"object-recognition","task_name":"Object Recognition"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2008.06043","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2008.06043"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eric-mitchell/macaw-min","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eric-mitchell/macaw","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":4},"by_repo_kind":{"listed":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"8cf7062eae489f4f","entry":"MLP","repo":"eric-mitchell/macaw-min","repo_kind":"listed","path":"impl.py","file_url":"https://github.com/eric-mitchell/macaw-min/blob/HEAD/impl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8cf7062eae489f4f"}},{"code_sha256_prefix":"b843875d4ee6d657","entry":"Experience","repo":"eric-mitchell/macaw-min","repo_kind":"listed","path":"impl.py","file_url":"https://github.com/eric-mitchell/macaw-min/blob/HEAD/impl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b843875d4ee6d657"}},{"code_sha256_prefix":"02080cb75a3041f8","entry":"ReplayBuffer","repo":"eric-mitchell/macaw-min","repo_kind":"listed","path":"impl.py","file_url":"https://github.com/eric-mitchell/macaw-min/blob/HEAD/impl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"02080cb75a3041f8"}},{"code_sha256_prefix":"17c803772d6eef45","entry":"WLinear","repo":"eric-mitchell/macaw-min","repo_kind":"listed","path":"impl.py","file_url":"https://github.com/eric-mitchell/macaw-min/blob/HEAD/impl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17c803772d6eef45"}},{"code_sha256_prefix":"c9acd7f503e226f6","entry":"build_networks_and_buffers","repo":"eric-mitchell/macaw-min","repo_kind":"listed","path":"impl.py","file_url":"https://github.com/eric-mitchell/macaw-min/blob/HEAD/impl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c9acd7f503e226f6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}