{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/policy-regularization-with-dataset-constraint","title":"Policy Regularization with Dataset Constraint for Offline Reinforcement Learning","arxiv_id":"2306.06569","date":"2023-06-11","proceeding":null,"authors":["Yuhang Ran","Yi-Chen Li","Fuxiang Zhang","Zongzhang Zhang","Yang Yu"],"abstract":"We consider the problem of learning the best possible policy from a fixed dataset, known as offline Reinforcement Learning (RL). A common taxonomy of existing offline RL works is policy regularization, which typically constrains the learned policy by distribution or support of the behavior policy. However, distribution and support constraints are overly conservative since they both force the policy to choose similar actions as the behavior policy when considering particular states. It will limit the learned policy's performance, especially when the behavior policy is sub-optimal. In this paper, we find that regularizing the policy towards the nearest state-action pair can be more effective and thus propose Policy Regularization with Dataset Constraint (PRDC). When updating the policy in a given state, PRDC searches the entire dataset for the nearest state-action sample and then restricts the policy with the action of this sample. Unlike previous works, PRDC can guide the policy with proper behaviors from the dataset, allowing it to choose actions that do not appear in the dataset along with the given state. It is a softer constraint but still keeps enough conservatism from out-of-distribution actions. Empirical evidence and theoretical analysis show that PRDC can alleviate offline RL's fundamentally challenging value overestimation issue with a bounded performance gap. Moreover, on a set of locomotion and navigation tasks, PRDC achieves state-of-the-art performance compared with existing methods. Code is available at https://github.com/LAMDA-RL/PRDC","url_abs":"https://arxiv.org/abs/2306.06569v2","url_pdf":"https://arxiv.org/pdf/2306.06569v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"policy-regularization-with-dataset-constraint","repo_url":"https://github.com/lamda-rl/prdc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"policy-regularization-with-dataset-constraint","repo_url":"https://github.com/zzmtsvv/rl_task","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.06569","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.06569"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LAMDA-RL/PRDC","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lamda-rl/prdc","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zzmtsvv/rl_task","reach":null}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"b6bad7ca8ac45549","entry":"Actor","repo":"lamda-rl/prdc","repo_kind":"official","path":"prdc.py","file_url":"https://github.com/lamda-rl/prdc/blob/HEAD/prdc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6bad7ca8ac45549"}},{"code_sha256_prefix":"d2ea31a27fb25f0e","entry":"Critic","repo":"lamda-rl/prdc","repo_kind":"official","path":"prdc.py","file_url":"https://github.com/lamda-rl/prdc/blob/HEAD/prdc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2ea31a27fb25f0e"}},{"code_sha256_prefix":"2f74230ac6a38e54","entry":"DeterministicActor","repo":"zzmtsvv/rl_task","repo_kind":"listed","path":"prdc/prdc.py","file_url":"https://github.com/zzmtsvv/rl_task/blob/HEAD/prdc/prdc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f74230ac6a38e54"}},{"code_sha256_prefix":"b2cd1bd32fbfa06a","entry":"EnsembledCritic","repo":"zzmtsvv/rl_task","repo_kind":"listed","path":"prdc/prdc.py","file_url":"https://github.com/zzmtsvv/rl_task/blob/HEAD/prdc/prdc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2cd1bd32fbfa06a"}},{"code_sha256_prefix":"44c3caba9cb900f9","entry":"PRDC","repo":"lamda-rl/prdc","repo_kind":"official","path":"prdc.py","file_url":"https://github.com/lamda-rl/prdc/blob/HEAD/prdc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"44c3caba9cb900f9"}},{"code_sha256_prefix":"3ea3ccde0c2a3214","entry":"prdc_config","repo":"zzmtsvv/rl_task","repo_kind":"listed","path":"prdc/prdc.py","file_url":"https://github.com/zzmtsvv/rl_task/blob/HEAD/prdc/prdc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ea3ccde0c2a3214"}},{"code_sha256_prefix":"ffb2596d71df2c51","entry":"PRDC","repo":"zzmtsvv/rl_task","repo_kind":"listed","path":"prdc/prdc.py","file_url":"https://github.com/zzmtsvv/rl_task/blob/HEAD/prdc/prdc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ffb2596d71df2c51"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}