{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/who-is-the-strongest-enemy-towards-optimal","title":"Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL","arxiv_id":"2106.05087","date":"2021-06-09","proceeding":"ICLR 2022 4","authors":["Yanchao Sun","Ruijie Zheng","Yongyuan Liang","Furong Huang"],"abstract":"Evaluating the worst-case performance of a reinforcement learning (RL) agent under the strongest/optimal adversarial perturbations on state observations (within some constraints) is crucial for understanding the robustness of RL agents. However, finding the optimal adversary is challenging, in terms of both whether we can find the optimal attack and how efficiently we can find it. Existing works on adversarial RL either use heuristics-based methods that may not find the strongest adversary, or directly train an RL-based adversary by treating the agent as a part of the environment, which can find the optimal adversary but may become intractable in a large state space. This paper introduces a novel attacking method to find the optimal attacks through collaboration between a designed function named \"actor\" and an RL-based learner named \"director\". The actor crafts state perturbations for a given policy perturbation direction, and the director learns to propose the best policy perturbation directions. Our proposed algorithm, PA-AD, is theoretically optimal and significantly more efficient than prior RL-based works in environments with large state spaces. Empirical results show that our proposed PA-AD universally outperforms state-of-the-art attacking methods in various Atari and MuJoCo environments. By applying PA-AD to adversarial training, we achieve state-of-the-art empirical robustness in multiple tasks under strong adversaries. The codebase is released at https://github.com/umd-huang-lab/paad_adv_rl.","url_abs":"https://arxiv.org/abs/2106.05087v5","url_pdf":"https://arxiv.org/pdf/2106.05087v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"who-is-the-strongest-enemy-towards-optimal","repo_url":"https://github.com/umd-huang-lab/paad_adv_rl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2106.05087","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.05087"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/umd-huang-lab/paad","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/umd-huang-lab/paad_adv_rl","reach":null}],"summary":{"ran_draft_wrong":4,"ran":2,"ran_fixture":2,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":13,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"a1c75c1d902ef7d5","entry":"common_fgsm","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/attacker/attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/attacker/attacker.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1c75c1d902ef7d5"}},{"code_sha256_prefix":"90d3059238cd3942","entry":"common_momentum_fgm","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/attacker/attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/attacker/attacker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90d3059238cd3942"}},{"code_sha256_prefix":"b37b0f21185a06e5","entry":"common_pgd","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/attacker/attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/attacker/attacker.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b37b0f21185a06e5"}},{"code_sha256_prefix":"bf1679c31b6bfadf","entry":"get_policy","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/trainer_adv/a2c_pa_attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/trainer_adv/a2c_pa_attacker.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf1679c31b6bfadf"}},{"code_sha256_prefix":"295c953ae1aedc03","entry":"noisy_pgd","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/attacker/attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/attacker/attacker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"295c953ae1aedc03"}},{"code_sha256_prefix":"21db8c7539b82cae","entry":"obs_dir_perturb_fgsm","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/trainer_adv/a2c_pa_attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/trainer_adv/a2c_pa_attacker.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"21db8c7539b82cae"}},{"code_sha256_prefix":"4aac02bd3c5c658b","entry":"obs_dir_perturb_momentum","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/trainer_adv/a2c_pa_attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/trainer_adv/a2c_pa_attacker.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4aac02bd3c5c658b"}},{"code_sha256_prefix":"b1e1cdc58f3fa6f6","entry":"reward2tensor","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/attack_robust/radial_train_attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/attack_robust/radial_train_attacker.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1e1cdc58f3fa6f6"}},{"code_sha256_prefix":"6517e45afce28963","entry":"state2tensor","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/attack_robust/radial_train_attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/attack_robust/radial_train_attacker.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6517e45afce28963"}},{"code_sha256_prefix":"942887ec27de2b72","entry":"Attacker","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/attacker/attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/attacker/attacker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"942887ec27de2b72"}},{"code_sha256_prefix":"b3ecb431fa1b4ac3","entry":"KL_Attack","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/attacker/attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/attacker/attacker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3ecb431fa1b4ac3"}},{"code_sha256_prefix":"a6f05c07b53af7b1","entry":"Param","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/attacker/attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/attacker/attacker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a6f05c07b53af7b1"}},{"code_sha256_prefix":"2e86d0eab58b3a7b","entry":"get_action","repo":"umd-huang-lab/paad_adv_rl","repo_kind":"official","path":"code_atari/paad_rl/attack_robust/radial_train_attacker.py","file_url":"https://github.com/umd-huang-lab/paad_adv_rl/blob/HEAD/code_atari/paad_rl/attack_robust/radial_train_attacker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e86d0eab58b3a7b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}