{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/task-agnostic-continual-reinforcement","title":"Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challenges","arxiv_id":"2205.14495","date":"2022-05-28","proceeding":null,"authors":["Massimo Caccia","Jonas Mueller","Taesup Kim","Laurent Charlin","Rasool Fakoor"],"abstract":"Continual learning (CL) enables the development of models and agents that learn from a sequence of tasks while addressing the limitations of standard deep learning approaches, such as catastrophic forgetting. In this work, we investigate the factors that contribute to the performance differences between task-agnostic CL and multi-task (MTL) agents. We pose two hypotheses: (1) task-agnostic methods might provide advantages in settings with limited data, computation, or high dimensionality, and (2) faster adaptation may be particularly beneficial in continual learning settings, helping to mitigate the effects of catastrophic forgetting. To investigate these hypotheses, we introduce a replay-based recurrent reinforcement learning (3RL) methodology for task-agnostic CL agents. We assess 3RL on a synthetic task and the Meta-World benchmark, which includes 50 unique manipulation tasks. Our results demonstrate that 3RL outperforms baseline methods and can even surpass its multi-task equivalent in challenging settings with high dimensionality. We also show that the recurrent task-agnostic agent consistently outperforms or matches the performance of its transformer-based counterpart. These findings provide insights into the advantages of task-agnostic CL over task-aware MTL approaches and highlight the potential of task-agnostic methods in resource-constrained, high-dimensional, and multi-task environments.","url_abs":"https://arxiv.org/abs/2205.14495v3","url_pdf":"https://arxiv.org/pdf/2205.14495v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"task-agnostic-continual-reinforcement","repo_url":"https://github.com/amazon-science/replay-based-recurrent-rl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"task-agnostic-continual-reinforcement","repo_url":"https://github.com/amazon-research/replay-based-recurrent-rl","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"experience-replay","method_name":"Experience Replay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.14495","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.14495"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-research/replay-based-recurrent-rl","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-science/replay-based-recurrent-rl","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"unverified":9},"by_repo_kind":{"official":{"samples":11,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d5f71e934c65e691","entry":"is_supported","repo":"amazon-research/replay-based-recurrent-rl","repo_kind":"official","path":"code/misc/autograd_hacks.py","file_url":"https://github.com/amazon-research/replay-based-recurrent-rl/blob/HEAD/code/misc/autograd_hacks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d5f71e934c65e691"}},{"code_sha256_prefix":"98948fae5be7e92e","entry":"profile","repo":"amazon-research/replay-based-recurrent-rl","repo_kind":"official","path":"code/oailibs/logger.py","file_url":"https://github.com/amazon-research/replay-based-recurrent-rl/blob/HEAD/code/oailibs/logger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98948fae5be7e92e"}},{"code_sha256_prefix":"c96673bfd86f55b0","entry":"atanh","repo":"amazon-research/replay-based-recurrent-rl","repo_kind":"official","path":"code/misc/utils.py","file_url":"https://github.com/amazon-research/replay-based-recurrent-rl/blob/HEAD/code/misc/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c96673bfd86f55b0"}},{"code_sha256_prefix":"94f234b2c98314c2","entry":"get_envs","repo":"amazon-research/replay-based-recurrent-rl","repo_kind":"official","path":"code/misc/sequoia_envs.py","file_url":"https://github.com/amazon-research/replay-based-recurrent-rl/blob/HEAD/code/misc/sequoia_envs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94f234b2c98314c2"}},{"code_sha256_prefix":"a61f2ad7df535ba0","entry":"get_state","repo":"amazon-research/replay-based-recurrent-rl","repo_kind":"official","path":"code/misc/torch_utility.py","file_url":"https://github.com/amazon-research/replay-based-recurrent-rl/blob/HEAD/code/misc/torch_utility.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a61f2ad7df535ba0"}},{"code_sha256_prefix":"b610b56e4fe7749d","entry":"hess_analysis","repo":"amazon-research/replay-based-recurrent-rl","repo_kind":"official","path":"code/misc/utils.py","file_url":"https://github.com/amazon-research/replay-based-recurrent-rl/blob/HEAD/code/misc/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b610b56e4fe7749d"}},{"code_sha256_prefix":"60105fbae1120564","entry":"load_model_states","repo":"amazon-research/replay-based-recurrent-rl","repo_kind":"official","path":"code/misc/torch_utility.py","file_url":"https://github.com/amazon-research/replay-based-recurrent-rl/blob/HEAD/code/misc/torch_utility.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60105fbae1120564"}},{"code_sha256_prefix":"c9637f11d7c5f378","entry":"make_output_format","repo":"amazon-research/replay-based-recurrent-rl","repo_kind":"official","path":"code/oailibs/logger.py","file_url":"https://github.com/amazon-research/replay-based-recurrent-rl/blob/HEAD/code/oailibs/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9637f11d7c5f378"}},{"code_sha256_prefix":"0493eaba8ac6cfcc","entry":"print_model_info","repo":"amazon-research/replay-based-recurrent-rl","repo_kind":"official","path":"code/misc/utils.py","file_url":"https://github.com/amazon-research/replay-based-recurrent-rl/blob/HEAD/code/misc/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0493eaba8ac6cfcc"}},{"code_sha256_prefix":"88ef5a17fc986d9a","entry":"read_json","repo":"amazon-research/replay-based-recurrent-rl","repo_kind":"official","path":"code/oailibs/logger.py","file_url":"https://github.com/amazon-research/replay-based-recurrent-rl/blob/HEAD/code/oailibs/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88ef5a17fc986d9a"}},{"code_sha256_prefix":"56afcb3496c8fd91","entry":"symsqrt","repo":"amazon-research/replay-based-recurrent-rl","repo_kind":"official","path":"code/misc/autograd_hacks.py","file_url":"https://github.com/amazon-research/replay-based-recurrent-rl/blob/HEAD/code/misc/autograd_hacks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56afcb3496c8fd91"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}