{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dr-sac-distributionally-robust-soft-actor","title":"DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertainty","arxiv_id":"2506.12622","date":"2025-06-14","proceeding":null,"authors":["Mingxuan Cui","Duo Zhou","Yuxuan Han","Grani A. Hanasusanto","Qiong Wang","huan zhang","Zhengyuan Zhou"],"abstract":"Deep reinforcement learning (RL) has achieved significant success, yet its application in real-world scenarios is often hindered by a lack of robustness to environmental uncertainties. To solve this challenge, some robust RL algorithms have been proposed, but most are limited to tabular settings. In this work, we propose Distributionally Robust Soft Actor-Critic (DR-SAC), a novel algorithm designed to enhance the robustness of the state-of-the-art Soft Actor-Critic (SAC) algorithm. DR-SAC aims to maximize the expected value with entropy against the worst possible transition model lying in an uncertainty set. A distributionally robust version of the soft policy iteration is derived with a convergence guarantee. For settings where nominal distributions are unknown, such as offline RL, a generative modeling approach is proposed to estimate the required nominal distributions from data. Furthermore, experimental results on a range of continuous control benchmark tasks demonstrate our algorithm achieves up to $9.8$ times the average reward of the SAC baseline under common perturbations. Additionally, compared with existing robust reinforcement learning algorithms, DR-SAC significantly improves computing efficiency and applicability to large-scale problems.","url_abs":"https://arxiv.org/abs/2506.12622v1","url_pdf":"https://arxiv.org/pdf/2506.12622v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dr-sac-distributionally-robust-soft-actor","repo_url":"https://github.com/lemutisme/dr-sac","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2506.12622","atlas_url":"https://app.syntology.ai/?focus=2506.12622","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.12622"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zaiyan-x/RFQI","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Lemutisme/DR-SAC","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lemutisme/dr-sac","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":9,"ran_honours":1,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":16,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b3d3944663449c1a","entry":"Actor","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3d3944663449c1a"}},{"code_sha256_prefix":"0a44e3fb45c1c563","entry":"ConditionalFlow","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a44e3fb45c1c563"}},{"code_sha256_prefix":"f7a6c367bf781d9e","entry":"Double_Q_Critic","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7a6c367bf781d9e"}},{"code_sha256_prefix":"7cb17db081d21d11","entry":"ExpActivation","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cb17db081d21d11"}},{"code_sha256_prefix":"0b400e7c6cd4e03d","entry":"MLPTransitionVAE","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b400e7c6cd4e03d"}},{"code_sha256_prefix":"1f773658fe04f371","entry":"Reward_adapter","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f773658fe04f371"}},{"code_sha256_prefix":"9922874222e5181e","entry":"SigmaMLP","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9922874222e5181e"}},{"code_sha256_prefix":"0479c4e4d9929e23","entry":"TimeMLP","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0479c4e4d9929e23"}},{"code_sha256_prefix":"0810f0b60522aad2","entry":"V_Critic","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0810f0b60522aad2"}},{"code_sha256_prefix":"89cefec7ee3edbdf","entry":"VectorizedCritic","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89cefec7ee3edbdf"}},{"code_sha256_prefix":"eae1a13a0cda9f0e","entry":"build_net","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eae1a13a0cda9f0e"}},{"code_sha256_prefix":"e7338a5540ebf890","entry":"ConditionalScoreModel","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7338a5540ebf890"}},{"code_sha256_prefix":"45c5e6c336c21050","entry":"ReplayBuffer","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45c5e6c336c21050"}},{"code_sha256_prefix":"b5f5d1ab656acdad","entry":"SAC_continuous","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5f5d1ab656acdad"}},{"code_sha256_prefix":"8c2dd772be9bbb05","entry":"TransitionDiffusion","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c2dd772be9bbb05"}},{"code_sha256_prefix":"84beae2ea3129aa8","entry":"dual","repo":"lemutisme/dr-sac","repo_kind":"official","path":"sac.py","file_url":"https://github.com/lemutisme/dr-sac/blob/HEAD/sac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84beae2ea3129aa8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}