{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffcps-diffusion-model-based-constrained","title":"DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning","arxiv_id":"2310.05333","date":"2023-10-09","proceeding":null,"authors":["Longxiang He","Li Shen","Linrui Zhang","Junbo Tan","Xueqian Wang"],"abstract":"Constrained policy search (CPS) is a fundamental problem in offline reinforcement learning, which is generally solved by advantage weighted regression (AWR). However, previous methods may still encounter out-of-distribution actions due to the limited expressivity of Gaussian-based policies. On the other hand, directly applying the state-of-the-art models with distribution expression capabilities (i.e., diffusion models) in the AWR framework is intractable since AWR requires exact policy probability densities, which is intractable in diffusion models. In this paper, we propose a novel approach, $\\textbf{Diffusion-based Constrained Policy Search}$ (dubbed DiffCPS), which tackles the diffusion-based constrained policy search with the primal-dual method. The theoretical analysis reveals that strong duality holds for diffusion-based CPS problems, and upon introducing parameter approximation, an approximated solution can be obtained after $\\mathcal{O}(1/\\epsilon)$ number of dual iterations, where $\\epsilon$ denotes the representation ability of the parametrized policy. Extensive experimental results based on the D4RL benchmark demonstrate the efficacy of our approach. We empirically show that DiffCPS achieves better or at least competitive performance compared to traditional AWR-based baselines as well as recent diffusion-based offline RL methods. The code is now available at https://github.com/felix-thu/DiffCPS.","url_abs":"https://arxiv.org/abs/2310.05333v2","url_pdf":"https://arxiv.org/pdf/2310.05333v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffcps-diffusion-model-based-constrained","repo_url":"https://github.com/felix-thu/DiffCPS","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.05333","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.05333"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/felix-thu/DiffCPS","reach":null}],"summary":{"ran":7,"ran_honours":2,"unverified":3},"by_repo_kind":{"official":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5bdc9227d5668ad4","entry":"Critic","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5bdc9227d5668ad4"}},{"code_sha256_prefix":"5c992de440a54f2a","entry":"EMA","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c992de440a54f2a"}},{"code_sha256_prefix":"cb1c89fe9b73e635","entry":"MLP","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb1c89fe9b73e635"}},{"code_sha256_prefix":"33d1a23a0c89bc5b","entry":"Progress","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"33d1a23a0c89bc5b"}},{"code_sha256_prefix":"98e65aa869e5b6c6","entry":"Silent","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98e65aa869e5b6c6"}},{"code_sha256_prefix":"a7e6cca0e2c2e381","entry":"WeightedL1","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7e6cca0e2c2e381"}},{"code_sha256_prefix":"1b472abd57f98492","entry":"WeightedL2","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b472abd57f98492"}},{"code_sha256_prefix":"63d21c7afce1a9e5","entry":"linear_beta_schedule","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"63d21c7afce1a9e5"}},{"code_sha256_prefix":"47c7ce0ace6e7c47","entry":"vp_beta_schedule","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"47c7ce0ace6e7c47"}},{"code_sha256_prefix":"c560690a1bed4e05","entry":"DiffCPS","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c560690a1bed4e05"}},{"code_sha256_prefix":"ead7aeb95e87333f","entry":"Diffusion","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ead7aeb95e87333f"}},{"code_sha256_prefix":"4a042f8f348b0f44","entry":"WeightedLoss","repo":"felix-thu/DiffCPS","repo_kind":"official","path":"agents/diffcps.py","file_url":"https://github.com/felix-thu/DiffCPS/blob/HEAD/agents/diffcps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a042f8f348b0f44"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}