{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-reinforcement-learn","title":"Learning to reinforcement learn","arxiv_id":"1611.05763","date":"2016-11-17","proceeding":null,"authors":["Jane. X. Wang","Zeb Kurth-Nelson","Dhruva Tirumala","Hubert Soyer","Joel Z. Leibo","Remi Munos","Charles Blundell","Dharshan Kumaran","Matt Botvinick"],"abstract":"In recent years deep reinforcement learning (RL) systems have attained\nsuperhuman performance in a number of challenging task domains. However, a\nmajor limitation of such applications is their demand for massive amounts of\ntraining data. A critical present objective is thus to develop deep RL methods\nthat can adapt rapidly to new tasks. In the present work we introduce a novel\napproach to this challenge, which we refer to as deep meta-reinforcement\nlearning. Previous work has shown that recurrent networks can support\nmeta-learning in a fully supervised context. We extend this approach to the RL\nsetting. What emerges is a system that is trained using one RL algorithm, but\nwhose recurrent dynamics implement a second, quite separate RL procedure. This\nsecond, learned RL algorithm can differ from the original one in arbitrary\nways. Importantly, because it is learned, it is configured to exploit structure\nin the training domain. We unpack these points in a series of seven\nproof-of-concept experiments, each of which examines a key aspect of deep\nmeta-RL. We consider prospects for extending and scaling up the approach, and\nalso point out some potentially important implications for neuroscience.","url_abs":"http://arxiv.org/abs/1611.05763v3","url_pdf":"http://arxiv.org/pdf/1611.05763v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-reinforcement-learn","repo_url":"https://github.com/Octavio-Pappalardo/RL2-implementation-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"learning-to-reinforcement-learn","repo_url":"https://github.com/Phutoast/Learning2RL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"learning-to-reinforcement-learn","repo_url":"https://github.com/RobvanGastel/meta-rl-algorithms","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-to-reinforcement-learn","repo_url":"https://github.com/awjuliani/Meta-RL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-to-reinforcement-learn","repo_url":"https://github.com/bkhmsi/meta-rl-harlow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"learning-to-reinforcement-learn","repo_url":"https://github.com/gomerudo/nas-dmrl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-to-reinforcement-learn","repo_url":"https://github.com/gomerudo/openai-baselines","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-to-reinforcement-learn","repo_url":"https://github.com/katalinic/l2rl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"learning-to-reinforcement-learn","repo_url":"https://github.com/mtrazzi/two-step-task","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"meta-reinforcement-learning","task_name":"Meta Reinforcement Learning"},{"task_slug":"meta-learning","task_name":"Meta-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1611.05763","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1611.05763"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gomerudo/openai-baselines","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Octavio-Pappalardo/RL2-implementation-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bkhmsi/meta-rl-harlow","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/katalinic/l2rl","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mtrazzi/two-step-task","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/awjuliani/Meta-RL","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gomerudo/nas-dmrl","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/RobvanGastel/meta-rl-algorithms","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Phutoast/Learning2RL","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":3,"unverified":12},"by_repo_kind":{"listed":{"samples":13,"ran":2,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"e01d3118f70d5b05","entry":"layer_init","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"e01d3118f70d5b05"}},{"code_sha256_prefix":"ef2d50dfca29c664","entry":"normalized_columns_initializer","repo":"awjuliani/Meta-RL","repo_kind":"listed","path":"helper.py","file_url":"https://github.com/awjuliani/Meta-RL/blob/HEAD/helper.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef2d50dfca29c664"}},{"code_sha256_prefix":"a44574392d168384","entry":"register","repo":"gomerudo/openai-baselines","repo_kind":"listed","path":"baselines/common/models.py","file_url":"https://github.com/gomerudo/openai-baselines/blob/HEAD/baselines/common/models.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a44574392d168384"}},{"code_sha256_prefix":"a3b4038f4e9b9479","entry":"cat_entropy","repo":"gomerudo/openai-baselines","repo_kind":"listed","path":"baselines/a2c/utils.py","file_url":"https://github.com/gomerudo/openai-baselines/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a3b4038f4e9b9479"}},{"code_sha256_prefix":"7082b4bbc0d9384f","entry":"cat_entropy_softmax","repo":"gomerudo/openai-baselines","repo_kind":"listed","path":"baselines/a2c/utils.py","file_url":"https://github.com/gomerudo/openai-baselines/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"7082b4bbc0d9384f"}},{"code_sha256_prefix":"a31c91d734c04428","entry":"collect_evaluation_data","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a31c91d734c04428"}},{"code_sha256_prefix":"fb210a4cf2658a85","entry":"discount","repo":"awjuliani/Meta-RL","repo_kind":"listed","path":"helper.py","file_url":"https://github.com/awjuliani/Meta-RL/blob/HEAD/helper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb210a4cf2658a85"}},{"code_sha256_prefix":"817d5caf0e3d3706","entry":"make_output_format","repo":"gomerudo/openai-baselines","repo_kind":"listed","path":"baselines/logger.py","file_url":"https://github.com/gomerudo/openai-baselines/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"817d5caf0e3d3706"}},{"code_sha256_prefix":"8455885a0d199802","entry":"profile","repo":"gomerudo/openai-baselines","repo_kind":"listed","path":"baselines/logger.py","file_url":"https://github.com/gomerudo/openai-baselines/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"8455885a0d199802"}},{"code_sha256_prefix":"88ef5a17fc986d9a","entry":"read_json","repo":"gomerudo/openai-baselines","repo_kind":"listed","path":"baselines/logger.py","file_url":"https://github.com/gomerudo/openai-baselines/blob/HEAD/baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"88ef5a17fc986d9a"}},{"code_sha256_prefix":"895ca4b776fb6d6d","entry":"rolling_window","repo":"gomerudo/openai-baselines","repo_kind":"listed","path":"baselines/results_plotter.py","file_url":"https://github.com/gomerudo/openai-baselines/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"895ca4b776fb6d6d"}},{"code_sha256_prefix":"178099b1f8fb2b9d","entry":"sample","repo":"gomerudo/openai-baselines","repo_kind":"listed","path":"baselines/a2c/utils.py","file_url":"https://github.com/gomerudo/openai-baselines/blob/HEAD/baselines/a2c/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"178099b1f8fb2b9d"}},{"code_sha256_prefix":"4ab4ad5ff9820e39","entry":"ts2xy","repo":"gomerudo/openai-baselines","repo_kind":"listed","path":"baselines/results_plotter.py","file_url":"https://github.com/gomerudo/openai-baselines/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"4ab4ad5ff9820e39"}},{"code_sha256_prefix":"5b304b9263d3a70a","entry":"update_target_graph","repo":"awjuliani/Meta-RL","repo_kind":"listed","path":"helper.py","file_url":"https://github.com/awjuliani/Meta-RL/blob/HEAD/helper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b304b9263d3a70a"}},{"code_sha256_prefix":"43741b9339bd80da","entry":"window_func","repo":"gomerudo/openai-baselines","repo_kind":"listed","path":"baselines/results_plotter.py","file_url":"https://github.com/gomerudo/openai-baselines/blob/HEAD/baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"43741b9339bd80da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}