{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adaptive-advantage-guided-policy","title":"Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning","arxiv_id":"2405.19909","date":"2024-05-30","proceeding":null,"authors":["Tenglong Liu","Yang Li","Yixing Lan","Hao Gao","Wei Pan","Xin Xu"],"abstract":"In offline reinforcement learning, the challenge of out-of-distribution (OOD) is pronounced. To address this, existing methods often constrain the learned policy through policy regularization. However, these methods often suffer from the issue of unnecessary conservativeness, hampering policy improvement. This occurs due to the indiscriminate use of all actions from the behavior policy that generates the offline dataset as constraints. The problem becomes particularly noticeable when the quality of the dataset is suboptimal. Thus, we propose Adaptive Advantage-guided Policy Regularization (A2PR), obtaining high-advantage actions from an augmented behavior policy combined with VAE to guide the learned policy. A2PR can select high-advantage actions that differ from those present in the dataset, while still effectively maintaining conservatism from OOD actions. This is achieved by harnessing the VAE capacity to generate samples matching the distribution of the data points. We theoretically prove that the improvement of the behavior policy is guaranteed. Besides, it effectively mitigates value overestimation with a bounded performance gap. Empirically, we conduct a series of experiments on the D4RL benchmark, where A2PR demonstrates state-of-the-art performance. Furthermore, experimental results on additional suboptimal mixed datasets reveal that A2PR exhibits superior performance. Code is available at https://github.com/ltlhuuu/A2PR.","url_abs":"https://arxiv.org/abs/2405.19909v3","url_pdf":"https://arxiv.org/pdf/2405.19909v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adaptive-advantage-guided-policy","repo_url":"https://github.com/ltlhuuu/a2pr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.19909","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.19909"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ltlhuuu/A2PR","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ltlhuuu/a2pr","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f879ba0e6ece5ebf","entry":"A2PR","repo":"ltlhuuu/a2pr","repo_kind":"official","path":"A2PR.py","file_url":"https://github.com/ltlhuuu/a2pr/blob/HEAD/A2PR.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f879ba0e6ece5ebf"}},{"code_sha256_prefix":"b550a7c7be5f130d","entry":"A2PR_Actor","repo":"ltlhuuu/a2pr","repo_kind":"official","path":"A2PR.py","file_url":"https://github.com/ltlhuuu/a2pr/blob/HEAD/A2PR.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b550a7c7be5f130d"}},{"code_sha256_prefix":"7daa282f895ac3b2","entry":"Critic","repo":"ltlhuuu/a2pr","repo_kind":"official","path":"A2PR.py","file_url":"https://github.com/ltlhuuu/a2pr/blob/HEAD/A2PR.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7daa282f895ac3b2"}},{"code_sha256_prefix":"d010e54b60df8f21","entry":"VAE","repo":"ltlhuuu/a2pr","repo_kind":"official","path":"A2PR.py","file_url":"https://github.com/ltlhuuu/a2pr/blob/HEAD/A2PR.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d010e54b60df8f21"}},{"code_sha256_prefix":"f135619af421a978","entry":"Value","repo":"ltlhuuu/a2pr","repo_kind":"official","path":"A2PR.py","file_url":"https://github.com/ltlhuuu/a2pr/blob/HEAD/A2PR.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f135619af421a978"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}