{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/model-based-safe-deep-reinforcement-learning","title":"Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm","arxiv_id":"2210.07573","date":"2022-10-14","proceeding":null,"authors":["Ashish Kumar Jayant","Shalabh Bhatnagar"],"abstract":"During initial iterations of training in most Reinforcement Learning (RL) algorithms, agents perform a significant number of random exploratory steps. In the real world, this can limit the practicality of these algorithms as it can lead to potentially dangerous behavior. Hence safe exploration is a critical issue in applying RL algorithms in the real world. This problem has been recently well studied under the Constrained Markov Decision Process (CMDP) Framework, where in addition to single-stage rewards, an agent receives single-stage costs or penalties as well depending on the state transitions. The prescribed cost functions are responsible for mapping undesirable behavior at any given time-step to a scalar value. The goal then is to find a feasible policy that maximizes reward returns while constraining the cost returns to be below a prescribed threshold during training as well as deployment. We propose an On-policy Model-based Safe Deep RL algorithm in which we learn the transition dynamics of the environment in an online manner as well as find a feasible optimal policy using the Lagrangian Relaxation-based Proximal Policy Optimization. We use an ensemble of neural networks with different initializations to tackle epistemic and aleatoric uncertainty issues faced during environment model learning. We compare our approach with relevant model-free and model-based approaches in Constrained RL using the challenging Safe Reinforcement Learning benchmark - the Open AI Safety Gym. We demonstrate that our algorithm is more sample efficient and results in lower cumulative hazard violations as compared to constrained model-free approaches. Further, our approach shows better reward performance than other constrained model-based approaches in the literature.","url_abs":"https://arxiv.org/abs/2210.07573v1","url_pdf":"https://arxiv.org/pdf/2210.07573v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"model-based-safe-deep-reinforcement-learning","repo_url":"https://github.com/akjayant/mbppol","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"safe-exploration","task_name":"Safe Exploration"},{"task_slug":"safe-reinforcement-learning","task_name":"Safe Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2210.07573","atlas_url":"https://app.syntology.ai/?focus=2210.07573","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.07573"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/akjayant/mbppol","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"official":{"samples":13,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7e36f1c357c4618d","entry":"colorize","repo":"akjayant/mbppol","repo_kind":"official","path":"src/utils/logx.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/utils/logx.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e36f1c357c4618d"}},{"code_sha256_prefix":"57542f87e248db45","entry":"combined_shape","repo":"akjayant/mbppol","repo_kind":"official","path":"src/core.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/core.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57542f87e248db45"}},{"code_sha256_prefix":"4cca5e460067b900","entry":"count_vars","repo":"akjayant/mbppol","repo_kind":"official","path":"src/core.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/core.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cca5e460067b900"}},{"code_sha256_prefix":"8a69cfc50e0f440c","entry":"dist_xy","repo":"akjayant/mbppol","repo_kind":"official","path":"src/aux.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/aux.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a69cfc50e0f440c"}},{"code_sha256_prefix":"f29202d8108bccd3","entry":"flat_concat","repo":"akjayant/mbppol","repo_kind":"official","path":"src/utils/mpi_tf.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/utils/mpi_tf.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f29202d8108bccd3"}},{"code_sha256_prefix":"37d85d44187b2915","entry":"get_data","repo":"akjayant/mbppol","repo_kind":"official","path":"src/model.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37d85d44187b2915"}},{"code_sha256_prefix":"6bd1e1acc559a3c0","entry":"get_goal_flag","repo":"akjayant/mbppol","repo_kind":"official","path":"src/aux.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/aux.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bd1e1acc559a3c0"}},{"code_sha256_prefix":"f09be98e3ec66db5","entry":"get_reward_cost","repo":"akjayant/mbppol","repo_kind":"official","path":"src/aux.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/aux.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f09be98e3ec66db5"}},{"code_sha256_prefix":"7569e2a512ec417c","entry":"mlp","repo":"akjayant/mbppol","repo_kind":"official","path":"src/core.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/core.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7569e2a512ec417c"}},{"code_sha256_prefix":"042875cf711accd0","entry":"assign_params_from_flat","repo":"akjayant/mbppol","repo_kind":"official","path":"src/utils/mpi_tf.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/utils/mpi_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"042875cf711accd0"}},{"code_sha256_prefix":"0661bf42848a4988","entry":"init_weights","repo":"akjayant/mbppol","repo_kind":"official","path":"src/model.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0661bf42848a4988"}},{"code_sha256_prefix":"8a70bbd1276e274c","entry":"restore_tf_graph","repo":"akjayant/mbppol","repo_kind":"official","path":"src/utils/logx.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/utils/logx.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a70bbd1276e274c"}},{"code_sha256_prefix":"82d7bab9b4ade8d1","entry":"sync_params","repo":"akjayant/mbppol","repo_kind":"official","path":"src/utils/mpi_tf.py","file_url":"https://github.com/akjayant/mbppol/blob/HEAD/src/utils/mpi_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82d7bab9b4ade8d1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}