{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-is-the-solution-for-state-adversarial","title":"What is the Solution for State-Adversarial Multi-Agent Reinforcement Learning?","arxiv_id":"2212.02705","date":"2022-12-06","proceeding":null,"authors":["Songyang Han","Sanbao Su","Sihong He","Shuo Han","Haizhao Yang","Shaofeng Zou","Fei Miao"],"abstract":"Various methods for Multi-Agent Reinforcement Learning (MARL) have been developed with the assumption that agents' policies are based on accurate state information. However, policies learned through Deep Reinforcement Learning (DRL) are susceptible to adversarial state perturbation attacks. In this work, we propose a State-Adversarial Markov Game (SAMG) and make the first attempt to investigate different solution concepts of MARL under state uncertainties. Our analysis shows that the commonly used solution concepts of optimal agent policy and robust Nash equilibrium do not always exist in SAMGs. To circumvent this difficulty, we consider a new solution concept called robust agent policy, where agents aim to maximize the worst-case expected state value. We prove the existence of robust agent policy for finite state and finite action SAMGs. Additionally, we propose a Robust Multi-Agent Adversarial Actor-Critic (RMA3C) algorithm to learn robust policies for MARL agents under state uncertainties. Our experiments demonstrate that our algorithm outperforms existing methods when faced with state perturbations and greatly improves the robustness of MARL policies. Our code is public on https://songyanghan.github.io/what_is_solution/.","url_abs":"https://arxiv.org/abs/2212.02705v5","url_pdf":"https://arxiv.org/pdf/2212.02705v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-is-the-solution-for-state-adversarial","repo_url":"https://github.com/susanbao/rmarl_code","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"multi-agent-reinforcement-learning","task_name":"Multi-agent Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2212.02705","atlas_url":"https://app.syntology.ai/?focus=2212.02705","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.02705"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/susanbao/rmarl_code","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_violates":1,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"d0a9e785ad9b047e","entry":"TruncatedNormal","repo":"susanbao/rmarl_code","repo_kind":"official","path":"rmarl/maddpg/trainer/maddpg_with_perturbed_networks.py","file_url":"https://github.com/susanbao/rmarl_code/blob/HEAD/rmarl/maddpg/trainer/maddpg_with_perturbed_networks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"d0a9e785ad9b047e"}},{"code_sha256_prefix":"fb985d688435bf65","entry":"discount_with_dones","repo":"susanbao/rmarl_code","repo_kind":"official","path":"rmarl/maddpg/trainer/maddpg_with_perturbed_networks.py","file_url":"https://github.com/susanbao/rmarl_code/blob/HEAD/rmarl/maddpg/trainer/maddpg_with_perturbed_networks.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"fb985d688435bf65"}},{"code_sha256_prefix":"a87ea3074224c0e8","entry":"make_env","repo":"susanbao/rmarl_code","repo_kind":"official","path":"multiagent-particle-envs/make_env.py","file_url":"https://github.com/susanbao/rmarl_code/blob/HEAD/multiagent-particle-envs/make_env.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"a87ea3074224c0e8"}},{"code_sha256_prefix":"aad5113d0b095522","entry":"shape_el","repo":"susanbao/rmarl_code","repo_kind":"official","path":"rmarl/maddpg/common/distributions.py","file_url":"https://github.com/susanbao/rmarl_code/blob/HEAD/rmarl/maddpg/common/distributions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"aad5113d0b095522"}},{"code_sha256_prefix":"24a950999549b85d","entry":"make_env","repo":"susanbao/rmarl_code","repo_kind":"official","path":"rmarl/experiments/train_with_perturbed_network.py","file_url":"https://github.com/susanbao/rmarl_code/blob/HEAD/rmarl/experiments/train_with_perturbed_network.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"24a950999549b85d"}},{"code_sha256_prefix":"8ebadf88476f8efe","entry":"mean","repo":"susanbao/rmarl_code","repo_kind":"official","path":"rmarl/maddpg/common/tf_util.py","file_url":"https://github.com/susanbao/rmarl_code/blob/HEAD/rmarl/maddpg/common/tf_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"8ebadf88476f8efe"}},{"code_sha256_prefix":"fbab233112bf4d1e","entry":"sum","repo":"susanbao/rmarl_code","repo_kind":"official","path":"rmarl/maddpg/common/tf_util.py","file_url":"https://github.com/susanbao/rmarl_code/blob/HEAD/rmarl/maddpg/common/tf_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"fbab233112bf4d1e"}},{"code_sha256_prefix":"598184a2689d36d0","entry":"var","repo":"susanbao/rmarl_code","repo_kind":"official","path":"rmarl/maddpg/common/tf_util.py","file_url":"https://github.com/susanbao/rmarl_code/blob/HEAD/rmarl/maddpg/common/tf_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"598184a2689d36d0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}