{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/safe-exploration-in-finite-markov-decision","title":"Safe Exploration in Finite Markov Decision Processes with Gaussian Processes","arxiv_id":"1606.04753","date":"2016-06-15","proceeding":"NeurIPS 2016 12","authors":["Matteo Turchetta","Felix Berkenkamp","Andreas Krause"],"abstract":"In classical reinforcement learning, when exploring an environment, agents\naccept arbitrary short term loss for long term gain. This is infeasible for\nsafety critical applications, such as robotics, where even a single unsafe\naction may cause system failure. In this paper, we address the problem of\nsafely exploring finite Markov decision processes (MDP). We define safety in\nterms of an, a priori unknown, safety constraint that depends on states and\nactions. We aim to explore the MDP under this constraint, assuming that the\nunknown function satisfies regularity conditions expressed via a Gaussian\nprocess prior. We develop a novel algorithm for this task and prove that it is\nable to completely explore the safely reachable part of the MDP without\nviolating the safety constraint. To achieve this, it cautiously explores safe\nstates and actions in order to gain statistical confidence about the safety of\nunvisited state-action pairs from noisy observations collected while navigating\nthe environment. Moreover, the algorithm explicitly considers reachability when\nexploring the MDP, ensuring that it does not get stuck in any state with no\nsafe way out. We demonstrate our method on digital terrain models for the task\nof exploring an unknown map with a rover.","url_abs":"http://arxiv.org/abs/1606.04753v2","url_pdf":"http://arxiv.org/pdf/1606.04753v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"safe-exploration-in-finite-markov-decision","repo_url":"https://github.com/befelix/SafeMDP","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"gaussian-processes","task_name":"Gaussian Processes"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"safe-exploration","task_name":"Safe Exploration"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1606.04753","atlas_url":"https://app.syntology.ai/?focus=1606.04753","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1606.04753"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/befelix/SafeMDP","reach":null}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"37fd50d8b81c8d3c","entry":"reachable_set","repo":"befelix/SafeMDP","repo_kind":"official","path":"safemdp/SafeMDP_class.py","file_url":"https://github.com/befelix/SafeMDP/blob/HEAD/safemdp/SafeMDP_class.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37fd50d8b81c8d3c"}},{"code_sha256_prefix":"0a331d10812517e0","entry":"returnable_set","repo":"befelix/SafeMDP","repo_kind":"official","path":"safemdp/SafeMDP_class.py","file_url":"https://github.com/befelix/SafeMDP/blob/HEAD/safemdp/SafeMDP_class.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a331d10812517e0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}