{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reachability-constrained-reinforcement","title":"Reachability Constrained Reinforcement Learning","arxiv_id":"2205.07536","date":"2022-05-16","proceeding":null,"authors":["Dongjie Yu","Haitong Ma","Shengbo Eben Li","Jianyu Chen"],"abstract":"Constrained reinforcement learning (CRL) has gained significant interest recently, since safety constraints satisfaction is critical for real-world problems. However, existing CRL methods constraining discounted cumulative costs generally lack rigorous definition and guarantee of safety. In contrast, in the safe control research, safety is defined as persistently satisfying certain state constraints. Such persistent safety is possible only on a subset of the state space, called feasible set, where an optimal largest feasible set exists for a given environment. Recent studies incorporate feasible sets into CRL with energy-based methods such as control barrier function (CBF), safety index (SI), and leverage prior conservative estimations of feasible sets, which harms the performance of the learned policy. To deal with this problem, this paper proposes the reachability CRL (RCRL) method by using reachability analysis to establish the novel self-consistency condition and characterize the feasible sets. The feasible sets are represented by the safety value function, which is used as the constraint in CRL. We use the multi-time scale stochastic approximation theory to prove that the proposed algorithm converges to a local optimum, where the largest feasible set can be guaranteed. Empirical results on different benchmarks validate the learned feasible set, the policy performance, and constraint satisfaction of RCRL, compared to CRL and safe control baselines.","url_abs":"https://arxiv.org/abs/2205.07536v2","url_pdf":"https://arxiv.org/pdf/2205.07536v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reachability-constrained-reinforcement","repo_url":"https://github.com/mahaitongdae/Reachability_Constrained_RL","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"reachability-constrained-reinforcement","repo_url":"https://github.com/mahaitongdae/safety_index_synthesis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"safe-reinforcement-learning","task_name":"Safe Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.07536","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.07536"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mahaitongdae/Reachability_Constrained_RL","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mahaitongdae/safety_index_synthesis","reach":null}],"summary":{"ran":3,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"ace8a2f6c849a951","entry":"Preprocessor","repo":"mahaitongdae/Reachability_Constrained_RL","repo_kind":"official","path":"learners/sac.py","file_url":"https://github.com/mahaitongdae/Reachability_Constrained_RL/blob/HEAD/learners/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ace8a2f6c849a951"}},{"code_sha256_prefix":"e4fcdf4a34ebca39","entry":"RunningMeanStd","repo":"mahaitongdae/Reachability_Constrained_RL","repo_kind":"official","path":"learners/sac.py","file_url":"https://github.com/mahaitongdae/Reachability_Constrained_RL/blob/HEAD/learners/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4fcdf4a34ebca39"}},{"code_sha256_prefix":"8c46617629ec6198","entry":"TimerStat","repo":"mahaitongdae/Reachability_Constrained_RL","repo_kind":"official","path":"learners/sac.py","file_url":"https://github.com/mahaitongdae/Reachability_Constrained_RL/blob/HEAD/learners/sac.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c46617629ec6198"}},{"code_sha256_prefix":"3ebb420a5d10cc64","entry":"update_mean_var_count_from_moments","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"3ebb420a5d10cc64"}},{"code_sha256_prefix":"df77743ef9174ca0","entry":"SACLearnerWithCost","repo":"mahaitongdae/Reachability_Constrained_RL","repo_kind":"official","path":"learners/sac.py","file_url":"https://github.com/mahaitongdae/Reachability_Constrained_RL/blob/HEAD/learners/sac.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df77743ef9174ca0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}