{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/s-2-ac-energy-based-reinforcement-learning","title":"S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic","arxiv_id":"2405.00987","date":"2024-05-02","proceeding":null,"authors":["Safa Messaoud","Billel Mokeddem","Zhenghai Xue","Linsey Pang","Bo An","Haipeng Chen","Sanjay Chawla"],"abstract":"Learning expressive stochastic policies instead of deterministic ones has been proposed to achieve better stability, sample complexity, and robustness. Notably, in Maximum Entropy Reinforcement Learning (MaxEnt RL), the policy is modeled as an expressive Energy-Based Model (EBM) over the Q-values. However, this formulation requires the estimation of the entropy of such EBMs, which is an open problem. To address this, previous MaxEnt RL methods either implicitly estimate the entropy, resulting in high computational complexity and variance (SQL), or follow a variational inference procedure that fits simplified actor distributions (e.g., Gaussian) for tractability (SAC). We propose Stein Soft Actor-Critic (S$^2$AC), a MaxEnt RL algorithm that learns expressive policies without compromising efficiency. Specifically, S$^2$AC uses parameterized Stein Variational Gradient Descent (SVGD) as the underlying policy. We derive a closed-form expression of the entropy of such policies. Our formula is computationally efficient and only depends on first-order derivatives and vector products. Empirical results show that S$^2$AC yields more optimal solutions to the MaxEnt objective than SQL and SAC in the multi-goal environment, and outperforms SAC and SQL on the MuJoCo benchmark. Our code is available at: https://github.com/SafaMessaoud/S2AC-Energy-Based-RL-with-Stein-Soft-Actor-Critic","url_abs":"https://arxiv.org/abs/2405.00987v1","url_pdf":"https://arxiv.org/pdf/2405.00987v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"s-2-ac-energy-based-reinforcement-learning","repo_url":"https://github.com/safamessaoud/s2ac-energy-based-rl-with-stein-soft-actor-critic","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"variational-inference","task_name":"Variational Inference"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dilated-convolution","method_name":"Dilated Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"sac","method_name":"SAC"},{"method_slug":"variational-inference","method_name":"Variational Inference"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2405.00987","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.00987"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/safamessaoud/s2ac-energy-based-rl-with-stein-soft-actor-critic","reach":null}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"17048a85840493d2","entry":"MLPFunction","repo":"safamessaoud/s2ac-energy-based-rl-with-stein-soft-actor-critic","repo_kind":"official","path":"STAC/actors/actor_svgd.py","file_url":"https://github.com/safamessaoud/s2ac-energy-based-rl-with-stein-soft-actor-critic/blob/HEAD/STAC/actors/actor_svgd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17048a85840493d2"}},{"code_sha256_prefix":"92f68d489e806c8d","entry":"MLPSquashedGaussian","repo":"safamessaoud/s2ac-energy-based-rl-with-stein-soft-actor-critic","repo_kind":"official","path":"STAC/actors/actor_svgd.py","file_url":"https://github.com/safamessaoud/s2ac-energy-based-rl-with-stein-soft-actor-critic/blob/HEAD/STAC/actors/actor_svgd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92f68d489e806c8d"}},{"code_sha256_prefix":"b6c9c8b3a7c87f1c","entry":"RBF","repo":"safamessaoud/s2ac-energy-based-rl-with-stein-soft-actor-critic","repo_kind":"official","path":"STAC/actors/actor_svgd.py","file_url":"https://github.com/safamessaoud/s2ac-energy-based-rl-with-stein-soft-actor-critic/blob/HEAD/STAC/actors/actor_svgd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6c9c8b3a7c87f1c"}},{"code_sha256_prefix":"22f6b89cab9ab447","entry":"ActorSvgd","repo":"safamessaoud/s2ac-energy-based-rl-with-stein-soft-actor-critic","repo_kind":"official","path":"STAC/actors/actor_svgd.py","file_url":"https://github.com/safamessaoud/s2ac-energy-based-rl-with-stein-soft-actor-critic/blob/HEAD/STAC/actors/actor_svgd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"22f6b89cab9ab447"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}