{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/recovery-rl-safe-reinforcement-learning-with","title":"Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones","arxiv_id":"2010.15920","date":"2020-10-29","proceeding":null,"authors":["Brijen Thananjeyan","Ashwin Balakrishna","Suraj Nair","Michael Luo","Krishnan Srinivasan","Minho Hwang","Joseph E. Gonzalez","Julian Ibarz","Chelsea Finn","Ken Goldberg"],"abstract":"Safety remains a central obstacle preventing widespread use of RL in the real world: learning new tasks in uncertain environments requires extensive exploration, but safety requires limiting exploration. We propose Recovery RL, an algorithm which navigates this tradeoff by (1) leveraging offline data to learn about constraint violating zones before policy learning and (2) separating the goals of improving task performance and constraint satisfaction across two policies: a task policy that only optimizes the task reward and a recovery policy that guides the agent to safety when constraint violation is likely. We evaluate Recovery RL on 6 simulation domains, including two contact-rich manipulation tasks and an image-based navigation task, and an image-based obstacle avoidance task on a physical robot. We compare Recovery RL to 5 prior safe RL methods which jointly optimize for task performance and safety via constrained optimization or reward shaping and find that Recovery RL outperforms the next best prior method across all domains. Results suggest that Recovery RL trades off constraint violations and task successes 2 - 20 times more efficiently in simulation domains and 3 times more efficiently in physical experiments. See https://tinyurl.com/rl-recovery for videos and supplementary material.","url_abs":"https://arxiv.org/abs/2010.15920v2","url_pdf":"https://arxiv.org/pdf/2010.15920v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"recovery-rl-safe-reinforcement-learning-with","repo_url":"https://github.com/abalakrishna123/recovery-rl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"recovery-rl-safe-reinforcement-learning-with","repo_url":"https://github.com/zlr20/saferl_kit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contact-rich-manipulation","task_name":"Contact-rich Manipulation"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"safe-reinforcement-learning","task_name":"Safe Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2010.15920","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.15920"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abalakrishna123/recovery-rl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zlr20/saferl_kit","reach":null}],"summary":{"ran_honours":2,"ran_draft_wrong":1,"ran_fixture":1},"by_repo_kind":{"listed":{"samples":4,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9380f9c95fc0dfaf","entry":"eval_policy","repo":"zlr20/saferl_kit","repo_kind":"listed","path":"saferl_algos/recovery.py","file_url":"https://github.com/zlr20/saferl_kit/blob/HEAD/saferl_algos/recovery.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9380f9c95fc0dfaf"}},{"code_sha256_prefix":"0559500297229526","entry":"get_stats","repo":"abalakrishna123/recovery-rl","repo_kind":"listed","path":"plotting/plot_runs.py","file_url":"https://github.com/abalakrishna123/recovery-rl/blob/HEAD/plotting/plot_runs.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0559500297229526"}},{"code_sha256_prefix":"b2c43b2ab933053b","entry":"moving_average","repo":"abalakrishna123/recovery-rl","repo_kind":"listed","path":"plotting/plot_runs.py","file_url":"https://github.com/abalakrishna123/recovery-rl/blob/HEAD/plotting/plot_runs.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b2c43b2ab933053b"}},{"code_sha256_prefix":"8c881242a42896d3","entry":"torchify","repo":"abalakrishna123/recovery-rl","repo_kind":"listed","path":"recovery_rl/experiment.py","file_url":"https://github.com/abalakrishna123/recovery-rl/blob/HEAD/recovery_rl/experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c881242a42896d3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}