{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/state-wise-safe-reinforcement-learning-with","title":"State-Wise Safe Reinforcement Learning With Pixel Observations","arxiv_id":"2311.02227","date":"2023-11-03","proceeding":null,"authors":["Simon Sinong Zhan","YiXuan Wang","Qingyuan Wu","Ruochen Jiao","Chao Huang","Qi Zhu"],"abstract":"In the context of safe exploration, Reinforcement Learning (RL) has long grappled with the challenges of balancing the tradeoff between maximizing rewards and minimizing safety violations, particularly in complex environments with contact-rich or non-smooth dynamics, and when dealing with high-dimensional pixel observations. Furthermore, incorporating state-wise safety constraints in the exploration and learning process, where the agent must avoid unsafe regions without prior knowledge, adds another layer of complexity. In this paper, we propose a novel pixel-observation safe RL algorithm that efficiently encodes state-wise safety constraints with unknown hazard regions through a newly introduced latent barrier-like function learning mechanism. As a joint learning framework, our approach begins by constructing a latent dynamics model with low-dimensional latent spaces derived from pixel observations. We then build and learn a latent barrier-like function on top of the latent dynamics and conduct policy optimization simultaneously, thereby improving both safety and the total expected return. Experimental evaluations on the safety-gym benchmark suite demonstrate that our proposed method significantly reduces safety violations throughout the training process, and demonstrates faster safety convergence compared to existing methods while achieving competitive results in reward return.","url_abs":"https://arxiv.org/abs/2311.02227v2","url_pdf":"https://arxiv.org/pdf/2311.02227v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"state-wise-safe-reinforcement-learning-with","repo_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"safe-exploration","task_name":"Safe Exploration"},{"task_slug":"safe-reinforcement-learning","task_name":"Safe Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.02227","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.02227"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/simonzhan-code/step-wise_saferl_pixel","reach":{"status":"ok"}}],"summary":{"ran":12,"unverified":2},"by_repo_kind":{"official":{"samples":14,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"cd4df7e37f410cf0","entry":"ObservationModel","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"modules/models.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/modules/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd4df7e37f410cf0"}},{"code_sha256_prefix":"a303dbe40ba89e74","entry":"get_lambda_datasets","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"utils/plot.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/utils/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a303dbe40ba89e74"}},{"code_sha256_prefix":"aa1afc09634f499a","entry":"get_model_free_baseline","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"utils/plot.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/utils/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa1afc09634f499a"}},{"code_sha256_prefix":"59c5b8b9c3d163d4","entry":"get_slac_datasets","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"utils/plot.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/utils/plot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"59c5b8b9c3d163d4"}},{"code_sha256_prefix":"529627f92815aafb","entry":"get_through_NN","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"modules/models.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/modules/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"529627f92815aafb"}},{"code_sha256_prefix":"f837e22509ce6a46","entry":"imagine_ahead","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f837e22509ce6a46"}},{"code_sha256_prefix":"390ab719c56fc08b","entry":"lambda_return","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"390ab719c56fc08b"}},{"code_sha256_prefix":"d7740644bf4b7de1","entry":"normalize","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"safety-gym/safety_gym/bench/bench_utils.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/safety-gym/safety_gym/bench/bench_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d7740644bf4b7de1"}},{"code_sha256_prefix":"278220aaad589e8c","entry":"postprocess_observation","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"modules/env.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/modules/env.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"278220aaad589e8c"}},{"code_sha256_prefix":"4ac8031d738ec401","entry":"quat2zalign","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"safety-gym/safety_gym/envs/engine.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/safety-gym/safety_gym/envs/engine.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ac8031d738ec401"}},{"code_sha256_prefix":"398af0fe872ccef0","entry":"reach_avoid_value_return","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"398af0fe872ccef0"}},{"code_sha256_prefix":"39708d407531ef32","entry":"theta2vec","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"safety-gym/safety_gym/envs/engine.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/safety-gym/safety_gym/envs/engine.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39708d407531ef32"}},{"code_sha256_prefix":"703763e9ba3cdadd","entry":"Env","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"modules/env.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/modules/env.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"703763e9ba3cdadd"}},{"code_sha256_prefix":"25c79a6d1db5ea05","entry":"bottle","repo":"simonzhan-code/step-wise_saferl_pixel","repo_kind":"official","path":"modules/models.py","file_url":"https://github.com/simonzhan-code/step-wise_saferl_pixel/blob/HEAD/modules/models.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"25c79a6d1db5ea05"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}