{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/locally-constrained-policy-optimization-for","title":"Online Reinforcement Learning in Non-Stationary Context-Driven Environments","arxiv_id":"2302.02182","date":"2023-02-04","proceeding":null,"authors":["Pouya Hamadanian","Arash Nasr-Esfahany","Malte Schwarzkopf","Siddartha Sen","Mohammad Alizadeh"],"abstract":"We study online reinforcement learning (RL) in non-stationary environments, where a time-varying exogenous context process affects the environment dynamics. Online RL is challenging in such environments due to \"catastrophic forgetting\" (CF). The agent tends to forget prior knowledge as it trains on new experiences. Prior approaches to mitigate this issue assume task labels (which are often not available in practice), employ brittle regularization heuristics, or use off-policy methods that suffer from instability and poor performance. We present Locally Constrained Policy Optimization (LCPO), an online RL approach that combats CF by anchoring policy outputs on old experiences while optimizing the return on current experiences. To perform this anchoring, LCPO locally constrains policy optimization using samples from experiences that lie outside of the current context distribution. We evaluate LCPO in Mujoco, classic control and computer systems environments with a variety of synthetic and real context traces, and find that it outperforms a variety of baselines in the non-stationary setting, while achieving results on-par with a \"prescient\" agent trained offline across all context traces. LCPO's source code is available at https://github.com/pouyahmdn/LCPO.","url_abs":"https://arxiv.org/abs/2302.02182v4","url_pdf":"https://arxiv.org/pdf/2302.02182v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"locally-constrained-policy-optimization-for","repo_url":"https://github.com/lcpo-rl/lcpo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"locally-constrained-policy-optimization-for","repo_url":"https://github.com/pouyahmdn/lcpo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2302.02182","atlas_url":"https://app.syntology.ai/?focus=2302.02182","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.02182"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/pouyahmdn/LCPO","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pouyahmdn/lcpo","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lcpo-rl/lcpo","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"7457c5d2d63ba05e","entry":"get_qu","repo":"pouyahmdn/lcpo","repo_kind":"official","path":"toy_grid_world/agent/core_alg/core_lcpo.py","file_url":"https://github.com/pouyahmdn/lcpo/blob/HEAD/toy_grid_world/agent/core_alg/core_lcpo.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7457c5d2d63ba05e"}},{"code_sha256_prefix":"a3e646f895ce380c","entry":"make_func","repo":"lcpo-rl/lcpo","repo_kind":"official","path":"straggler_mitigate/agent/lcpo.py","file_url":"https://github.com/lcpo-rl/lcpo/blob/HEAD/straggler_mitigate/agent/lcpo.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a3e646f895ce380c"}},{"code_sha256_prefix":"1c2bd33bc18c7920","entry":"mlp_seq","repo":"lcpo-rl/lcpo","repo_kind":"official","path":"disc-gym/neural_net/nn.py","file_url":"https://github.com/lcpo-rl/lcpo/blob/HEAD/disc-gym/neural_net/nn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"1c2bd33bc18c7920"}},{"code_sha256_prefix":"9caa532b1a67ea69","entry":"t","repo":"lcpo-rl/lcpo","repo_kind":"official","path":"disc-gym/plot_all_figures.py","file_url":"https://github.com/lcpo-rl/lcpo/blob/HEAD/disc-gym/plot_all_figures.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"9caa532b1a67ea69"}},{"code_sha256_prefix":"d0400323aff44d84","entry":"handle_jobs","repo":"lcpo-rl/lcpo","repo_kind":"official","path":"disc-gym/launch_multi_exp.py","file_url":"https://github.com/lcpo-rl/lcpo/blob/HEAD/disc-gym/launch_multi_exp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"d0400323aff44d84"}},{"code_sha256_prefix":"523e3f740ce0914d","entry":"handle_jobs","repo":"lcpo-rl/lcpo","repo_kind":"official","path":"straggler_mitigate/launch_multi_exp.py","file_url":"https://github.com/lcpo-rl/lcpo/blob/HEAD/straggler_mitigate/launch_multi_exp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"523e3f740ce0914d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}