{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/robust-reinforcement-learning-on-state-1","title":"Robust Reinforcement Learning on State Observations with Learned Optimal Adversary","arxiv_id":"2101.08452","date":"2021-01-21","proceeding":"ICLR 2021 1","authors":["huan zhang","Hongge Chen","Duane Boning","Cho-Jui Hsieh"],"abstract":"We study the robustness of reinforcement learning (RL) with adversarially perturbed state observations, which aligns with the setting of many adversarial attacks to deep reinforcement learning (DRL) and is also important for rolling out real-world RL agent under unpredictable sensing noise. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found, which is guaranteed to obtain the worst case agent reward. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones. To enhance the robustness of an agent, we propose a framework of alternating training with learned adversaries (ATLA), which trains an adversary online together with the agent using policy gradient following the optimal adversarial attack framework. Additionally, inspired by the analysis of state-adversarial Markov decision process (SA-MDP), we show that past states and actions (history) can be useful for learning a robust agent, and we empirically find a LSTM based policy can be more robust under adversaries. Empirical evaluations on a few continuous control environments show that ATLA achieves state-of-the-art performance under strong adversaries. Our code is available at https://github.com/huanzhang12/ATLA_robust_RL.","url_abs":"https://arxiv.org/abs/2101.08452v1","url_pdf":"https://arxiv.org/pdf/2101.08452v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"robust-reinforcement-learning-on-state-1","repo_url":"https://github.com/huanzhang12/ATLA_robust_RL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"robust-reinforcement-learning-on-state-1","repo_url":"https://github.com/huanzhang12/sa_ppo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-attack","task_name":"Adversarial Attack"},{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2101.08452","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2101.08452"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/huanzhang12/ATLA_robust_RL","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huanzhang12/sa_ppo","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"listed":{"samples":8,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"59236ac3123afcdf","entry":"adv_normalize","repo":"huanzhang12/sa_ppo","repo_kind":"listed","path":"src/policy_gradients/steps.py","file_url":"https://github.com/huanzhang12/sa_ppo/blob/HEAD/src/policy_gradients/steps.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"59236ac3123afcdf"}},{"code_sha256_prefix":"cc9596016142c080","entry":"get_state_kl_bound_sgld","repo":"huanzhang12/sa_ppo","repo_kind":"listed","path":"src/policy_gradients/steps.py","file_url":"https://github.com/huanzhang12/sa_ppo/blob/HEAD/src/policy_gradients/steps.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc9596016142c080"}},{"code_sha256_prefix":"fe36ea6d910477c3","entry":"pack_history","repo":"huanzhang12/sa_ppo","repo_kind":"listed","path":"src/policy_gradients/steps.py","file_url":"https://github.com/huanzhang12/sa_ppo/blob/HEAD/src/policy_gradients/steps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe36ea6d910477c3"}},{"code_sha256_prefix":"08d7633cda3a7602","entry":"robust_ppo_step","repo":"huanzhang12/sa_ppo","repo_kind":"listed","path":"src/policy_gradients/steps.py","file_url":"https://github.com/huanzhang12/sa_ppo/blob/HEAD/src/policy_gradients/steps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"08d7633cda3a7602"}},{"code_sha256_prefix":"428b336da9f35ed2","entry":"surrogate_reward","repo":"huanzhang12/sa_ppo","repo_kind":"listed","path":"src/policy_gradients/steps.py","file_url":"https://github.com/huanzhang12/sa_ppo/blob/HEAD/src/policy_gradients/steps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"428b336da9f35ed2"}},{"code_sha256_prefix":"e4187dd121d7ca14","entry":"value_loss_gae","repo":"huanzhang12/sa_ppo","repo_kind":"listed","path":"src/policy_gradients/steps.py","file_url":"https://github.com/huanzhang12/sa_ppo/blob/HEAD/src/policy_gradients/steps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4187dd121d7ca14"}},{"code_sha256_prefix":"3e4c2bb78b75b63a","entry":"value_loss_returns","repo":"huanzhang12/sa_ppo","repo_kind":"listed","path":"src/policy_gradients/steps.py","file_url":"https://github.com/huanzhang12/sa_ppo/blob/HEAD/src/policy_gradients/steps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e4c2bb78b75b63a"}},{"code_sha256_prefix":"368a7f7b219ba34e","entry":"value_step","repo":"huanzhang12/sa_ppo","repo_kind":"listed","path":"src/policy_gradients/steps.py","file_url":"https://github.com/huanzhang12/sa_ppo/blob/HEAD/src/policy_gradients/steps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"368a7f7b219ba34e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}