{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/order-matters-agent-by-agent-policy","title":"Order Matters: Agent-by-agent Policy Optimization","arxiv_id":"2302.06205","date":"2023-02-13","proceeding":null,"authors":["Xihuai Wang","Zheng Tian","Ziyu Wan","Ying Wen","Jun Wang","Weinan Zhang"],"abstract":"While multi-agent trust region algorithms have achieved great success empirically in solving coordination tasks, most of them, however, suffer from a non-stationarity problem since agents update their policies simultaneously. In contrast, a sequential scheme that updates policies agent-by-agent provides another perspective and shows strong performance. However, sample inefficiency and lack of monotonic improvement guarantees for each agent are still the two significant challenges for the sequential scheme. In this paper, we propose the \\textbf{A}gent-by-\\textbf{a}gent \\textbf{P}olicy \\textbf{O}ptimization (A2PO) algorithm to improve the sample efficiency and retain the guarantees of monotonic improvement for each agent during training. We justify the tightness of the monotonic improvement bound compared with other trust region algorithms. From the perspective of sequentially updating agents, we further consider the effect of agent updating order and extend the theory of non-stationarity into the sequential update scheme. To evaluate A2PO, we conduct a comprehensive empirical study on four benchmarks: StarCraftII, Multi-agent MuJoCo, Multi-agent Particle Environment, and Google Research Football full game scenarios. A2PO consistently outperforms strong baselines.","url_abs":"https://arxiv.org/abs/2302.06205v2","url_pdf":"https://arxiv.org/pdf/2302.06205v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"order-matters-agent-by-agent-policy","repo_url":"https://github.com/xihuai18/A2PO-ICLR2023","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"mujoco","task_name":"MuJoCo"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"biggan","method_name":"BigGAN"},{"method_slug":"conditional-batch-normalization","method_name":"Conditional Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"early-stopping","method_name":"Early Stopping"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"gan-hinge-loss","method_name":"GAN Hinge Loss"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"non-local-block","method_name":"Non-Local Block"},{"method_slug":"non-local-operation","method_name":"Non-Local Operation"},{"method_slug":"off-diagonal-orthogonal-regularization","method_name":"Off-Diagonal Orthogonal Regularization"},{"method_slug":"projection-discriminator","method_name":"Projection Discriminator"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sagan","method_name":"SAGAN"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"spectral-normalization","method_name":"Spectral Normalization"},{"method_slug":"ttur","method_name":"TTUR"},{"method_slug":"truncation-trick","method_name":"Truncation Trick"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2302.06205","atlas_url":"https://app.syntology.ai/?focus=2302.06205","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.06205"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xihuai18/A2PO-ICLR2023","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9589eca5bae9de80","entry":"check","repo":"xihuai18/A2PO-ICLR2023","repo_kind":"official","path":"onpolicy/algorithms/utils/util.py","file_url":"https://github.com/xihuai18/A2PO-ICLR2023/blob/HEAD/onpolicy/algorithms/utils/util.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9589eca5bae9de80"}},{"code_sha256_prefix":"9de355e93051e4ad","entry":"init","repo":"xihuai18/A2PO-ICLR2023","repo_kind":"official","path":"onpolicy/algorithms/utils/util.py","file_url":"https://github.com/xihuai18/A2PO-ICLR2023/blob/HEAD/onpolicy/algorithms/utils/util.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9de355e93051e4ad"}},{"code_sha256_prefix":"2cbb027c3ce7bc4e","entry":"near_linear","repo":"xihuai18/A2PO-ICLR2023","repo_kind":"official","path":"onpolicy/algorithms/utils/tune_constant.py","file_url":"https://github.com/xihuai18/A2PO-ICLR2023/blob/HEAD/onpolicy/algorithms/utils/tune_constant.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2cbb027c3ce7bc4e"}},{"code_sha256_prefix":"0944240e80ec7625","entry":"get_clones","repo":"xihuai18/A2PO-ICLR2023","repo_kind":"official","path":"onpolicy/algorithms/utils/util.py","file_url":"https://github.com/xihuai18/A2PO-ICLR2023/blob/HEAD/onpolicy/algorithms/utils/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0944240e80ec7625"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}