{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beyond-worst-case-attacks-robust-rl-with","title":"Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies","arxiv_id":"2402.12673","date":"2024-02-20","proceeding":null,"authors":["Xiangyu Liu","ChengHao Deng","Yanchao Sun","Yongyuan Liang","Furong Huang"],"abstract":"In light of the burgeoning success of reinforcement learning (RL) in diverse real-world applications, considerable focus has been directed towards ensuring RL policies are robust to adversarial attacks during test time. Current approaches largely revolve around solving a minimax problem to prepare for potential worst-case scenarios. While effective against strong attacks, these methods often compromise performance in the absence of attacks or the presence of only weak attacks. To address this, we study policy robustness under the well-accepted state-adversarial attack model, extending our focus beyond only worst-case attacks. We first formalize this task at test time as a regret minimization problem and establish its intrinsic hardness in achieving sublinear regret when the baseline policy is from a general continuous policy class, $\\Pi$. This finding prompts us to \\textit{refine} the baseline policy class $\\Pi$ prior to test time, aiming for efficient adaptation within a finite policy class $\\Tilde{\\Pi}$, which can resort to an adversarial bandit subroutine. In light of the importance of a small, finite $\\Tilde{\\Pi}$, we propose a novel training-time algorithm to iteratively discover \\textit{non-dominated policies}, forming a near-optimal and minimal $\\Tilde{\\Pi}$, thereby ensuring both robustness and test-time efficiency. Empirical validation on the Mujoco corroborates the superiority of our approach in terms of natural and robust performance, as well as adaptability to various attack scenarios.","url_abs":"https://arxiv.org/abs/2402.12673v1","url_pdf":"https://arxiv.org/pdf/2402.12673v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beyond-worst-case-attacks-robust-rl-with","repo_url":"https://github.com/umd-huang-lab/protected","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-attack","task_name":"Adversarial Attack"},{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.12673","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.12673"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/umd-huang-lab/protected","reach":null}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"a41b550a8ff5fe84","entry":"robust_ppo_step","repo":"umd-huang-lab/protected","repo_kind":"official","path":"policy_gradients/steps.py","file_url":"https://github.com/umd-huang-lab/protected/blob/HEAD/policy_gradients/steps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a41b550a8ff5fe84"}},{"code_sha256_prefix":"ceb1c5e7ed63f591","entry":"value_step","repo":"umd-huang-lab/protected","repo_kind":"official","path":"policy_gradients/steps.py","file_url":"https://github.com/umd-huang-lab/protected/blob/HEAD/policy_gradients/steps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ceb1c5e7ed63f591"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}