{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffusion-actor-critic-formulating","title":"Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning","arxiv_id":"2405.20555","date":"2024-05-31","proceeding":null,"authors":["Linjiajie Fang","Ruoxue Liu","Jing Zhang","Wenjia Wang","Bing-Yi Jing"],"abstract":"In offline reinforcement learning (RL), it is necessary to manage out-of-distribution actions to prevent overestimation of value functions. Policy-regularized methods address this problem by constraining the target policy to stay close to the behavior policy. Although several approaches suggest representing the behavior policy as an expressive diffusion model to boost performance, it remains unclear how to regularize the target policy given a diffusion-modeled behavior sampler. In this paper, we propose Diffusion Actor-Critic (DAC) that formulates the Kullback-Leibler (KL) constraint policy iteration as a diffusion noise regression problem, enabling direct representation of target policies as diffusion models. Our approach follows the actor-critic learning paradigm that we alternatively train a diffusion-modeled target policy and a critic network. The actor training loss includes a soft Q-guidance term from the Q-gradient. The soft Q-guidance grounds on the theoretical solution of the KL constraint policy iteration, which prevents the learned policy from taking out-of-distribution actions. For critic training, we train a Q-ensemble to stabilize the estimation of Q-gradient. Additionally, DAC employs lower confidence bound (LCB) to address the overestimation and underestimation of value targets due to function approximation error. Our approach is evaluated on the D4RL benchmarks and outperforms the state-of-the-art in almost all environments. Code is available at \\href{https://github.com/Fang-Lin93/DAC}{\\texttt{github.com/Fang-Lin93/DAC}}.","url_abs":"https://arxiv.org/abs/2405.20555v1","url_pdf":"https://arxiv.org/pdf/2405.20555v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffusion-actor-critic-formulating","repo_url":"https://github.com/Fang-Lin93/DAC","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"dac","method_name":"DAC"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.20555","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.20555"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Fang-Lin93/DAC","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran":7,"ran_honours":1},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"894ddcc18b0d0b2b","entry":"compare_curves","repo":"Fang-Lin93/DAC","repo_kind":"official","path":"plots.py","file_url":"https://github.com/Fang-Lin93/DAC/blob/HEAD/plots.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"894ddcc18b0d0b2b"}},{"code_sha256_prefix":"7aea8bb5ea8ef141","entry":"compare_hist","repo":"Fang-Lin93/DAC","repo_kind":"official","path":"plots.py","file_url":"https://github.com/Fang-Lin93/DAC/blob/HEAD/plots.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7aea8bb5ea8ef141"}},{"code_sha256_prefix":"159a09a700e91b2e","entry":"get_weight_decay_mask","repo":"Fang-Lin93/DAC","repo_kind":"official","path":"networks/model.py","file_url":"https://github.com/Fang-Lin93/DAC/blob/HEAD/networks/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"159a09a700e91b2e"}},{"code_sha256_prefix":"12ab4b77c5a0f513","entry":"get_weight_decay_mask_excludes","repo":"Fang-Lin93/DAC","repo_kind":"official","path":"networks/model.py","file_url":"https://github.com/Fang-Lin93/DAC/blob/HEAD/networks/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"12ab4b77c5a0f513"}},{"code_sha256_prefix":"7fc1d127bf81537f","entry":"mish","repo":"Fang-Lin93/DAC","repo_kind":"official","path":"agents/dac.py","file_url":"https://github.com/Fang-Lin93/DAC/blob/HEAD/agents/dac.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7fc1d127bf81537f"}},{"code_sha256_prefix":"87c04751d25ad6e2","entry":"orthogonal_init","repo":"Fang-Lin93/DAC","repo_kind":"official","path":"networks/initialization.py","file_url":"https://github.com/Fang-Lin93/DAC/blob/HEAD/networks/initialization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"87c04751d25ad6e2"}},{"code_sha256_prefix":"2966f3014a4193a5","entry":"plot_curve","repo":"Fang-Lin93/DAC","repo_kind":"official","path":"plots.py","file_url":"https://github.com/Fang-Lin93/DAC/blob/HEAD/plots.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2966f3014a4193a5"}},{"code_sha256_prefix":"a1e4cfcce3126d87","entry":"uniform_init","repo":"Fang-Lin93/DAC","repo_kind":"official","path":"networks/initialization.py","file_url":"https://github.com/Fang-Lin93/DAC/blob/HEAD/networks/initialization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a1e4cfcce3126d87"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}