{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/off-policy-primal-dual-safe-reinforcement","title":"Off-Policy Primal-Dual Safe Reinforcement Learning","arxiv_id":"2401.14758","date":"2024-01-26","proceeding":null,"authors":["Zifan Wu","Bo Tang","Qian Lin","Chao Yu","Shangqin Mao","Qianlong Xie","Xingxing Wang","Dong Wang"],"abstract":"Primal-dual safe RL methods commonly perform iterations between the primal update of the policy and the dual update of the Lagrange Multiplier. Such a training paradigm is highly susceptible to the error in cumulative cost estimation since this estimation serves as the key bond connecting the primal and dual update processes. We show that this problem causes significant underestimation of cost when using off-policy methods, leading to the failure to satisfy the safety constraint. To address this issue, we propose conservative policy optimization, which learns a policy in a constraint-satisfying area by considering the uncertainty in cost estimation. This improves constraint satisfaction but also potentially hinders reward maximization. We then introduce local policy convexification to help eliminate such suboptimality by gradually reducing the estimation uncertainty. We provide theoretical interpretations of the joint coupling effect of these two ingredients and further verify them by extensive experiments. Results on benchmark tasks show that our method not only achieves an asymptotic performance comparable to state-of-the-art on-policy methods while using much fewer samples, but also significantly reduces constraint violation during training. Our code is available at https://github.com/ZifanWu/CAL.","url_abs":"https://arxiv.org/abs/2401.14758v2","url_pdf":"https://arxiv.org/pdf/2401.14758v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"off-policy-primal-dual-safe-reinforcement","repo_url":"https://github.com/pku-alignment/omnisafe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"off-policy-primal-dual-safe-reinforcement","repo_url":"https://github.com/zifanwu/cal","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"safe-reinforcement-learning","task_name":"Safe Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.14758","atlas_url":"https://app.syntology.ai/?focus=2401.14758","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.14758"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ZifanWu/CAL","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/AlgTUDelft/WCSAC","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/liuzuxin/cvpo-safe-rl","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/zmsn-2077/CUP-safe-rl","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/PKU-Alignment/omnisafe","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/liuzuxin/fsrl","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zifanwu/cal","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pku-alignment/omnisafe","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":10,"ran_draft_wrong":1,"ran_fixture":1,"ran_violates":1,"unverified":9},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1},"found_in_text":{"samples":15,"ran":7,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"e47f49c341d79904","entry":"CUP","repo":"zmsn-2077/CUP-safe-rl","repo_kind":"found_in_text","path":"algorithm/cup.py","file_url":"https://github.com/zmsn-2077/CUP-safe-rl/blob/HEAD/algorithm/cup.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e47f49c341d79904"}},{"code_sha256_prefix":"7e36f1c357c4618d","entry":"colorize","repo":"AlgTUDelft/WCSAC","repo_kind":"found_in_text","path":"wc_sac/utils/logx.py","file_url":"https://github.com/AlgTUDelft/WCSAC/blob/HEAD/wc_sac/utils/logx.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e36f1c357c4618d"}},{"code_sha256_prefix":"fbc505a5dac82646","entry":"convert_json","repo":"AlgTUDelft/WCSAC","repo_kind":"found_in_text","path":"wc_sac/utils/serialization_utils.py","file_url":"https://github.com/AlgTUDelft/WCSAC/blob/HEAD/wc_sac/utils/serialization_utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbc505a5dac82646"}},{"code_sha256_prefix":"2526cfd8d5bf5d06","entry":"create_log_gaussian","repo":"ZifanWu/CAL","repo_kind":"official","path":"agent/utils.py","file_url":"https://github.com/ZifanWu/CAL/blob/HEAD/agent/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2526cfd8d5bf5d06"}},{"code_sha256_prefix":"f29202d8108bccd3","entry":"flat_concat","repo":"AlgTUDelft/WCSAC","repo_kind":"found_in_text","path":"wc_sac/utils/mpi_tf.py","file_url":"https://github.com/AlgTUDelft/WCSAC/blob/HEAD/wc_sac/utils/mpi_tf.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f29202d8108bccd3"}},{"code_sha256_prefix":"5b17e81927faf55e","entry":"gaussian_kl","repo":"zmsn-2077/CUP-safe-rl","repo_kind":"found_in_text","path":"algorithm/cup.py","file_url":"https://github.com/zmsn-2077/CUP-safe-rl/blob/HEAD/algorithm/cup.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b17e81927faf55e"}},{"code_sha256_prefix":"cb396863dfa04b08","entry":"get_threshold","repo":"ZifanWu/CAL","repo_kind":"official","path":"env/constraints.py","file_url":"https://github.com/ZifanWu/CAL/blob/HEAD/env/constraints.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb396863dfa04b08"}},{"code_sha256_prefix":"9ca8b8e45f7fbebf","entry":"is_json_serializable","repo":"AlgTUDelft/WCSAC","repo_kind":"found_in_text","path":"wc_sac/utils/serialization_utils.py","file_url":"https://github.com/AlgTUDelft/WCSAC/blob/HEAD/wc_sac/utils/serialization_utils.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ca8b8e45f7fbebf"}},{"code_sha256_prefix":"09556d56c8b73050","entry":"logsumexp","repo":"ZifanWu/CAL","repo_kind":"official","path":"agent/utils.py","file_url":"https://github.com/ZifanWu/CAL/blob/HEAD/agent/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09556d56c8b73050"}},{"code_sha256_prefix":"d7740644bf4b7de1","entry":"normalize","repo":"ZifanWu/CAL","repo_kind":"official","path":"env/safety-gym/safety_gym/bench/bench_utils.py","file_url":"https://github.com/ZifanWu/CAL/blob/HEAD/env/safety-gym/safety_gym/bench/bench_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7740644bf4b7de1"}},{"code_sha256_prefix":"4ac8031d738ec401","entry":"quat2zalign","repo":"ZifanWu/CAL","repo_kind":"official","path":"env/safety-gym/safety_gym/envs/engine.py","file_url":"https://github.com/ZifanWu/CAL/blob/HEAD/env/safety-gym/safety_gym/envs/engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ac8031d738ec401"}},{"code_sha256_prefix":"cc9f89f4007acda0","entry":"setup_logger_kwargs","repo":"AlgTUDelft/WCSAC","repo_kind":"found_in_text","path":"wc_sac/utils/run_utils.py","file_url":"https://github.com/AlgTUDelft/WCSAC/blob/HEAD/wc_sac/utils/run_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc9f89f4007acda0"}},{"code_sha256_prefix":"39708d407531ef32","entry":"theta2vec","repo":"ZifanWu/CAL","repo_kind":"official","path":"env/safety-gym/safety_gym/envs/engine.py","file_url":"https://github.com/ZifanWu/CAL/blob/HEAD/env/safety-gym/safety_gym/envs/engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39708d407531ef32"}},{"code_sha256_prefix":"042875cf711accd0","entry":"assign_params_from_flat","repo":"AlgTUDelft/WCSAC","repo_kind":"found_in_text","path":"wc_sac/utils/mpi_tf.py","file_url":"https://github.com/AlgTUDelft/WCSAC/blob/HEAD/wc_sac/utils/mpi_tf.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"042875cf711accd0"}},{"code_sha256_prefix":"a2580d7c4a763348","entry":"get_vars","repo":"AlgTUDelft/WCSAC","repo_kind":"found_in_text","path":"wc_sac/sac/wcsac.py","file_url":"https://github.com/AlgTUDelft/WCSAC/blob/HEAD/wc_sac/sac/wcsac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a2580d7c4a763348"}},{"code_sha256_prefix":"7d8c58ad49e14ee5","entry":"graph_detach","repo":"zmsn-2077/CUP-safe-rl","repo_kind":"found_in_text","path":"algorithm/cup.py","file_url":"https://github.com/zmsn-2077/CUP-safe-rl/blob/HEAD/algorithm/cup.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d8c58ad49e14ee5"}},{"code_sha256_prefix":"0661bf42848a4988","entry":"init_weights","repo":"ZifanWu/CAL","repo_kind":"official","path":"agent/model.py","file_url":"https://github.com/ZifanWu/CAL/blob/HEAD/agent/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0661bf42848a4988"}},{"code_sha256_prefix":"e6f12e62f88b74b9","entry":"mlp","repo":"AlgTUDelft/WCSAC","repo_kind":"found_in_text","path":"wc_sac/sac/wcsac.py","file_url":"https://github.com/AlgTUDelft/WCSAC/blob/HEAD/wc_sac/sac/wcsac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6f12e62f88b74b9"}},{"code_sha256_prefix":"17b03c462ce92576","entry":"placeholder","repo":"AlgTUDelft/WCSAC","repo_kind":"found_in_text","path":"wc_sac/sac/wcsac.py","file_url":"https://github.com/AlgTUDelft/WCSAC/blob/HEAD/wc_sac/sac/wcsac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17b03c462ce92576"}},{"code_sha256_prefix":"a3c2d567fb278fea","entry":"println","repo":"zmsn-2077/CUP-safe-rl","repo_kind":"found_in_text","path":"algorithm/cup.py","file_url":"https://github.com/zmsn-2077/CUP-safe-rl/blob/HEAD/algorithm/cup.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a3c2d567fb278fea"}},{"code_sha256_prefix":"006604b1b94a9fe7","entry":"restore_tf_graph","repo":"AlgTUDelft/WCSAC","repo_kind":"found_in_text","path":"wc_sac/utils/logx.py","file_url":"https://github.com/AlgTUDelft/WCSAC/blob/HEAD/wc_sac/utils/logx.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"006604b1b94a9fe7"}},{"code_sha256_prefix":"495130bd57c2f5c1","entry":"to_dytype_device","repo":"zmsn-2077/CUP-safe-rl","repo_kind":"found_in_text","path":"algorithm/cup.py","file_url":"https://github.com/zmsn-2077/CUP-safe-rl/blob/HEAD/algorithm/cup.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"495130bd57c2f5c1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}