{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffusion-actor-critic-with-entropy-regulator","title":"Diffusion Actor-Critic with Entropy Regulator","arxiv_id":"2405.15177","date":"2024-05-24","proceeding":null,"authors":["Yinuo Wang","Likun Wang","YuXuan Jiang","Wenjun Zou","Tong Liu","Xujie Song","Wenxuan Wang","Liming Xiao","Jiang Wu","Jingliang Duan","Shengbo Eben Li"],"abstract":"Reinforcement learning (RL) has proven highly effective in addressing complex decision-making and control tasks. However, in most traditional RL algorithms, the policy is typically parameterized as a diagonal Gaussian distribution with learned mean and variance, which constrains their capability to acquire complex policies. In response to this problem, we propose an online RL algorithm termed diffusion actor-critic with entropy regulator (DACER). This algorithm conceptualizes the reverse process of the diffusion model as a novel policy function and leverages the capability of the diffusion model to fit multimodal distributions, thereby enhancing the representational capacity of the policy. Since the distribution of the diffusion policy lacks an analytical expression, its entropy cannot be determined analytically. To mitigate this, we propose a method to estimate the entropy of the diffusion policy utilizing Gaussian mixture model. Building on the estimated entropy, we can learn a parameter $\\alpha$ that modulates the degree of exploration and exploitation. Parameter $\\alpha$ will be employed to adaptively regulate the variance of the added noise, which is applied to the action output by the diffusion model. Experimental trials on MuJoCo benchmarks and a multimodal task demonstrate that the DACER algorithm achieves state-of-the-art (SOTA) performance in most MuJoCo control tasks while exhibiting a stronger representational capacity of the diffusion policy.","url_abs":"https://arxiv.org/abs/2405.15177v5","url_pdf":"https://arxiv.org/pdf/2405.15177v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffusion-actor-critic-with-entropy-regulator","repo_url":"https://github.com/happy-yan/DACER-Diffusion-with-Online-RL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.15177","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.15177"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/happy-yan/DACER-Diffusion-with-Online-RL","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"a546de74aa434993","entry":"estimate_entropy","repo":"happy-yan/DACER-Diffusion-with-Online-RL","repo_kind":"official","path":"relax/algorithm/dacer.py","file_url":"https://github.com/happy-yan/DACER-Diffusion-with-Online-RL/blob/HEAD/relax/algorithm/dacer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a546de74aa434993"}},{"code_sha256_prefix":"ab957cbc27bee860","entry":"mlp","repo":"happy-yan/DACER-Diffusion-with-Online-RL","repo_kind":"official","path":"relax/network/blocks.py","file_url":"https://github.com/happy-yan/DACER-Diffusion-with-Online-RL/blob/HEAD/relax/network/blocks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab957cbc27bee860"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}