{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/robust-deep-reinforcement-learning-through-2","title":"Regret-Based Defense in Adversarial Reinforcement Learning","arxiv_id":"2302.06912","date":"2023-02-14","proceeding":null,"authors":["Roman Belaire","Pradeep Varakantham","Thanh Nguyen","David Lo"],"abstract":"Deep Reinforcement Learning (DRL) policies have been shown to be vulnerable to small adversarial noise in observations. Such adversarial noise can have disastrous consequences in safety-critical environments. For instance, a self-driving car receiving adversarially perturbed sensory observations about nearby signs (e.g., a stop sign physically altered to be perceived as a speed limit sign) or objects (e.g., cars altered to be recognized as trees) can be fatal. Existing approaches for making RL algorithms robust to an observation-perturbing adversary have focused on reactive approaches that iteratively improve against adversarial examples generated at each iteration. While such approaches have been shown to provide improvements over regular RL methods, they are reactive and can fare significantly worse if certain categories of adversarial examples are not generated during training. To that end, we pursue a more proactive approach that relies on directly optimizing a well-studied robustness measure, regret instead of expected value. We provide a principled approach that minimizes maximum regret over a \"neighborhood\" of observations to the received \"observation\". Our regret criterion can be used to modify existing value- and policy-based Deep RL methods. We demonstrate that our approaches provide a significant improvement in performance across a wide variety of benchmarks against leading approaches for robust Deep RL.","url_abs":"https://arxiv.org/abs/2302.06912v4","url_pdf":"https://arxiv.org/pdf/2302.06912v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"robust-deep-reinforcement-learning-through-2","repo_url":"https://github.com/romanbelaire/robust-ccer","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.06912","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.06912"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/romanbelaire/robust-ccer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran":2,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"35b6f17122388f57","entry":"dict_product","repo":"romanbelaire/robust-ccer","repo_kind":"official","path":"code_mujoco/ccer/utils.py","file_url":"https://github.com/romanbelaire/robust-ccer/blob/HEAD/code_mujoco/ccer/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35b6f17122388f57"}},{"code_sha256_prefix":"85bf3d86b586cab7","entry":"get_alg_name","repo":"romanbelaire/robust-ccer","repo_kind":"official","path":"code_mujoco/ccer/get_best_pickle.py","file_url":"https://github.com/romanbelaire/robust-ccer/blob/HEAD/code_mujoco/ccer/get_best_pickle.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85bf3d86b586cab7"}},{"code_sha256_prefix":"35a892ade6c313d2","entry":"get_env_name","repo":"romanbelaire/robust-ccer","repo_kind":"official","path":"code_mujoco/ccer/get_best_pickle.py","file_url":"https://github.com/romanbelaire/robust-ccer/blob/HEAD/code_mujoco/ccer/get_best_pickle.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35a892ade6c313d2"}},{"code_sha256_prefix":"942706632c22fc44","entry":"iwt","repo":"romanbelaire/robust-ccer","repo_kind":"official","path":"code_mujoco/ccer/utils.py","file_url":"https://github.com/romanbelaire/robust-ccer/blob/HEAD/code_mujoco/ccer/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"942706632c22fc44"}},{"code_sha256_prefix":"61098cd5c9624563","entry":"partialclass","repo":"romanbelaire/robust-ccer","repo_kind":"official","path":"code_mujoco/ccer/policy_gradients/models.py","file_url":"https://github.com/romanbelaire/robust-ccer/blob/HEAD/code_mujoco/ccer/policy_gradients/models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61098cd5c9624563"}},{"code_sha256_prefix":"075dfe30db346a5d","entry":"consolidate_experiment","repo":"romanbelaire/robust-ccer","repo_kind":"official","path":"code_mujoco/ccer/cox/generator.py","file_url":"https://github.com/romanbelaire/robust-ccer/blob/HEAD/code_mujoco/ccer/cox/generator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"075dfe30db346a5d"}},{"code_sha256_prefix":"b0c1eddd4ae7590e","entry":"generate_experiments","repo":"romanbelaire/robust-ccer","repo_kind":"official","path":"code_mujoco/ccer/cox/generator.py","file_url":"https://github.com/romanbelaire/robust-ccer/blob/HEAD/code_mujoco/ccer/cox/generator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0c1eddd4ae7590e"}},{"code_sha256_prefix":"cd38a29fe01f4be3","entry":"has_tensorboard","repo":"romanbelaire/robust-ccer","repo_kind":"official","path":"code_mujoco/ccer/cox/archive.py","file_url":"https://github.com/romanbelaire/robust-ccer/blob/HEAD/code_mujoco/ccer/cox/archive.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd38a29fe01f4be3"}},{"code_sha256_prefix":"934a0a9624814d4e","entry":"pack_history","repo":"romanbelaire/robust-ccer","repo_kind":"official","path":"code_mujoco/ccer/policy_gradients/models.py","file_url":"https://github.com/romanbelaire/robust-ccer/blob/HEAD/code_mujoco/ccer/policy_gradients/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"934a0a9624814d4e"}},{"code_sha256_prefix":"cfbb5cb3d22968c8","entry":"schema_from_dict","repo":"romanbelaire/robust-ccer","repo_kind":"official","path":"code_mujoco/ccer/cox/store.py","file_url":"https://github.com/romanbelaire/robust-ccer/blob/HEAD/code_mujoco/ccer/cox/store.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfbb5cb3d22968c8"}},{"code_sha256_prefix":"0c2504cab96329f1","entry":"unpack_history","repo":"romanbelaire/robust-ccer","repo_kind":"official","path":"code_mujoco/ccer/policy_gradients/models.py","file_url":"https://github.com/romanbelaire/robust-ccer/blob/HEAD/code_mujoco/ccer/policy_gradients/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c2504cab96329f1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}