{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/maximum-entropy-reinforcement-learning-via","title":"Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow","arxiv_id":"2405.13629","date":"2024-05-22","proceeding":null,"authors":["Chen-Hao Chao","Chien Feng","Wei-Fang Sun","Cheng-Kuang Lee","Simon See","Chun-Yi Lee"],"abstract":"Existing Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) methods for continuous action spaces are typically formulated based on actor-critic frameworks and optimized through alternating steps of policy evaluation and policy improvement. In the policy evaluation steps, the critic is updated to capture the soft Q-function. In the policy improvement steps, the actor is adjusted in accordance with the updated soft Q-function. In this paper, we introduce a new MaxEnt RL framework modeled using Energy-Based Normalizing Flows (EBFlow). This framework integrates the policy evaluation steps and the policy improvement steps, resulting in a single objective training process. Our method enables the calculation of the soft value function used in the policy evaluation target without Monte Carlo approximation. Moreover, this design supports the modeling of multi-modal action distributions while facilitating efficient action sampling. To evaluate the performance of our method, we conducted experiments on the MuJoCo benchmark suite and a number of high-dimensional robotic tasks simulated by Omniverse Isaac Gym. The evaluation results demonstrate that our method achieves superior performance compared to widely-adopted representative baselines.","url_abs":"https://arxiv.org/abs/2405.13629v2","url_pdf":"https://arxiv.org/pdf/2405.13629v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"maximum-entropy-reinforcement-learning-via","repo_url":"https://github.com/ChienFeng-hub/meow","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"ingenuity","task_name":"Ingenuity"},{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"omniverse-isaac-gym","task_name":"Omniverse Isaac Gym"},{"task_slug":"openai-gym","task_name":"OpenAI Gym"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"normalizing-flows","method_name":"Normalizing Flows"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/openai-gym-on-ant-v4","task":"OpenAI Gym","dataset":"Ant-v4","model":"MEow","rank_in_archive_order":1,"of":5,"metrics":{"Average Return":"6586.33"},"uses_additional_data":false},{"leaderboard":"/sota/openai-gym-on-halfcheetah-v4","task":"OpenAI Gym","dataset":"HalfCheetah-v4","model":"MEow","rank_in_archive_order":4,"of":5,"metrics":{"Average Return":"10981.47"},"uses_additional_data":false},{"leaderboard":"/sota/openai-gym-on-hopper-v4","task":"OpenAI Gym","dataset":"Hopper-v4","model":"MEow","rank_in_archive_order":1,"of":5,"metrics":{"Average Return":"3332.99"},"uses_additional_data":false},{"leaderboard":"/sota/openai-gym-on-humanoid-v4","task":"OpenAI Gym","dataset":"Humanoid-v4","model":"MEow","rank_in_archive_order":1,"of":5,"metrics":{"Average Return":"6923.22"},"uses_additional_data":false},{"leaderboard":"/sota/openai-gym-on-walker2d-v4","task":"OpenAI Gym","dataset":"Walker2d-v4","model":"MEow","rank_in_archive_order":2,"of":5,"metrics":{"Average Return":"5526.66"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.13629","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}