{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ompo-a-unified-framework-for-rl-under-policy","title":"OMPO: A Unified Framework for RL under Policy and Dynamics Shifts","arxiv_id":"2405.19080","date":"2024-05-29","proceeding":null,"authors":["Yu Luo","Tianying Ji","Fuchun Sun","Jianwei Zhang","Huazhe Xu","Xianyuan Zhan"],"abstract":"Training reinforcement learning policies using environment interaction data collected from varying policies or dynamics presents a fundamental challenge. Existing works often overlook the distribution discrepancies induced by policy or dynamics shifts, or rely on specialized algorithms with task priors, thus often resulting in suboptimal policy performances and high learning variances. In this paper, we identify a unified strategy for online RL policy learning under diverse settings of policy and dynamics shifts: transition occupancy matching. In light of this, we introduce a surrogate policy learning objective by considering the transition occupancy discrepancies and then cast it into a tractable min-max optimization problem through dual reformulation. Our method, dubbed Occupancy-Matching Policy Optimization (OMPO), features a specialized actor-critic structure equipped with a distribution discriminator and a small-size local buffer. We conduct extensive experiments based on the OpenAI Gym, Meta-World, and Panda Robots environments, encompassing policy shifts under stationary and nonstationary dynamics, as well as domain adaption. The results demonstrate that OMPO outperforms the specialized baselines from different categories in all settings. We also find that OMPO exhibits particularly strong performance when combined with domain randomization, highlighting its potential in RL-based robotics applications","url_abs":"https://arxiv.org/abs/2405.19080v1","url_pdf":"https://arxiv.org/pdf/2405.19080v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ompo-a-unified-framework-for-rl-under-policy","repo_url":"https://github.com/roythuly/ompo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"openai-gym","task_name":"OpenAI Gym"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.19080","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.19080"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/roythuly/ompo","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"2526cfd8d5bf5d06","entry":"create_log_gaussian","repo":"roythuly/ompo","repo_kind":"official","path":"model/utils.py","file_url":"https://github.com/roythuly/ompo/blob/HEAD/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2526cfd8d5bf5d06"}},{"code_sha256_prefix":"09556d56c8b73050","entry":"logsumexp","repo":"roythuly/ompo","repo_kind":"official","path":"model/utils.py","file_url":"https://github.com/roythuly/ompo/blob/HEAD/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09556d56c8b73050"}},{"code_sha256_prefix":"2b8b073e7cad0d69","entry":"orthogonal_regularization","repo":"roythuly/ompo","repo_kind":"official","path":"model/utils.py","file_url":"https://github.com/roythuly/ompo/blob/HEAD/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b8b073e7cad0d69"}},{"code_sha256_prefix":"81924566bfec4a85","entry":"panda_make_env","repo":"roythuly/ompo","repo_kind":"official","path":"Non_stationary_env/panda_env.py","file_url":"https://github.com/roythuly/ompo/blob/HEAD/Non_stationary_env/panda_env.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81924566bfec4a85"}},{"code_sha256_prefix":"30836a3e793a43ac","entry":"panda_make_noise_env","repo":"roythuly/ompo","repo_kind":"official","path":"Non_stationary_env/panda_env.py","file_url":"https://github.com/roythuly/ompo/blob/HEAD/Non_stationary_env/panda_env.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30836a3e793a43ac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}