{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-modulate-pre-trained-models-in-rl-1","title":"Learning to Modulate pre-trained Models in RL","arxiv_id":"2306.14884","date":"2023-06-26","proceeding":"NeurIPS 2023 11","authors":["Thomas Schmied","Markus Hofmarcher","Fabian Paischer","Razvan Pascanu","Sepp Hochreiter"],"abstract":"Reinforcement Learning (RL) has been successful in various domains like robotics, game playing, and simulation. While RL agents have shown impressive capabilities in their specific tasks, they insufficiently adapt to new tasks. In supervised learning, this adaptation problem is addressed by large-scale pre-training followed by fine-tuning to new down-stream tasks. Recently, pre-training on multiple tasks has been gaining traction in RL. However, fine-tuning a pre-trained model often suffers from catastrophic forgetting. That is, the performance on the pre-training tasks deteriorates when fine-tuning on new tasks. To investigate the catastrophic forgetting phenomenon, we first jointly pre-train a model on datasets from two benchmark suites, namely Meta-World and DMControl. Then, we evaluate and compare a variety of fine-tuning methods prevalent in natural language processing, both in terms of performance on new tasks, and how well performance on pre-training tasks is retained. Our study shows that with most fine-tuning approaches, the performance on pre-training tasks deteriorates significantly. Therefore, we propose a novel method, Learning-to-Modulate (L2M), that avoids the degradation of learned skills by modulating the information flow of the frozen pre-trained model via a learnable modulation pool. Our method achieves state-of-the-art performance on the Continual-World benchmark, while retaining performance on the pre-training tasks. Finally, to aid future research in this area, we release a dataset encompassing 50 Meta-World and 16 DMControl tasks.","url_abs":"https://arxiv.org/abs/2306.14884v2","url_pdf":"https://arxiv.org/pdf/2306.14884v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-modulate-pre-trained-models-in-rl-1","repo_url":"https://github.com/ml-jku/l2m","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.14884","atlas_url":"https://app.syntology.ai/?focus=2306.14884","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.14884"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-jku/l2m","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":9},"by_repo_kind":{"official":{"samples":10,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8ad77ab59e479221","entry":"get_param_count","repo":"ml-jku/l2m","repo_kind":"official","path":"src/algos/agent_utils.py","file_url":"https://github.com/ml-jku/l2m/blob/HEAD/src/algos/agent_utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ad77ab59e479221"}},{"code_sha256_prefix":"eea50b3088b40480","entry":"compute_rtg_from_target","repo":"ml-jku/l2m","repo_kind":"official","path":"src/buffers/buffer_utils.py","file_url":"https://github.com/ml-jku/l2m/blob/HEAD/src/buffers/buffer_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eea50b3088b40480"}},{"code_sha256_prefix":"0dea1b97816bb8f5","entry":"copy_params_dict","repo":"ml-jku/l2m","repo_kind":"official","path":"src/algos/decision_transformer_with_ewc_sb3.py","file_url":"https://github.com/ml-jku/l2m/blob/HEAD/src/algos/decision_transformer_with_ewc_sb3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0dea1b97816bb8f5"}},{"code_sha256_prefix":"47f553906dc6d889","entry":"discount_cumsum","repo":"ml-jku/l2m","repo_kind":"official","path":"src/buffers/buffer_utils.py","file_url":"https://github.com/ml-jku/l2m/blob/HEAD/src/buffers/buffer_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47f553906dc6d889"}},{"code_sha256_prefix":"561155d921a7e0b3","entry":"discount_cumsum_np","repo":"ml-jku/l2m","repo_kind":"official","path":"src/buffers/buffer_utils.py","file_url":"https://github.com/ml-jku/l2m/blob/HEAD/src/buffers/buffer_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"561155d921a7e0b3"}},{"code_sha256_prefix":"8357e0d1e97197da","entry":"make_augmentations","repo":"ml-jku/l2m","repo_kind":"official","path":"src/augmentations/augs.py","file_url":"https://github.com/ml-jku/l2m/blob/HEAD/src/augmentations/augs.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8357e0d1e97197da"}},{"code_sha256_prefix":"868522feaa56b737","entry":"make_loss_fn","repo":"ml-jku/l2m","repo_kind":"official","path":"src/algos/agent_utils.py","file_url":"https://github.com/ml-jku/l2m/blob/HEAD/src/algos/agent_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"868522feaa56b737"}},{"code_sha256_prefix":"5144b139b6efaac1","entry":"make_random_proj_matrix","repo":"ml-jku/l2m","repo_kind":"official","path":"src/algos/agent_utils.py","file_url":"https://github.com/ml-jku/l2m/blob/HEAD/src/algos/agent_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5144b139b6efaac1"}},{"code_sha256_prefix":"d14f15ef38ff3737","entry":"onelike_params_dict","repo":"ml-jku/l2m","repo_kind":"official","path":"src/algos/decision_transformer_with_ewc_sb3.py","file_url":"https://github.com/ml-jku/l2m/blob/HEAD/src/algos/decision_transformer_with_ewc_sb3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d14f15ef38ff3737"}},{"code_sha256_prefix":"0abec4a44be91ea1","entry":"zerolike_params_dict","repo":"ml-jku/l2m","repo_kind":"official","path":"src/algos/decision_transformer_with_ewc_sb3.py","file_url":"https://github.com/ml-jku/l2m/blob/HEAD/src/algos/decision_transformer_with_ewc_sb3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0abec4a44be91ea1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}