{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ace-cooperative-multi-agent-q-learning-with","title":"ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependency","arxiv_id":"2211.16068","date":"2022-11-29","proceeding":null,"authors":["Chuming Li","Jie Liu","Yinmin Zhang","Yuhong Wei","Yazhe Niu","Yaodong Yang","Yu Liu","Wanli Ouyang"],"abstract":"Multi-agent reinforcement learning (MARL) suffers from the non-stationarity problem, which is the ever-changing targets at every iteration when multiple agents update their policies at the same time. Starting from first principle, in this paper, we manage to solve the non-stationarity problem by proposing bidirectional action-dependent Q-learning (ACE). Central to the development of ACE is the sequential decision-making process wherein only one agent is allowed to take action at one time. Within this process, each agent maximizes its value function given the actions taken by the preceding agents at the inference stage. In the learning phase, each agent minimizes the TD error that is dependent on how the subsequent agents have reacted to their chosen action. Given the design of bidirectional dependency, ACE effectively turns a multiagent MDP into a single-agent MDP. We implement the ACE framework by identifying the proper network representation to formulate the action dependency, so that the sequential decision process is computed implicitly in one forward pass. To validate ACE, we compare it with strong baselines on two MARL benchmarks. Empirical experiments demonstrate that ACE outperforms the state-of-the-art algorithms on Google Research Football and StarCraft Multi-Agent Challenge by a large margin. In particular, on SMAC tasks, ACE achieves 100% success rate on almost all the hard and super-hard maps. We further study extensive research problems regarding ACE, including extension, generalization, and practicability. Code is made available to facilitate further research.","url_abs":"https://arxiv.org/abs/2211.16068v2","url_pdf":"https://arxiv.org/pdf/2211.16068v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ace-cooperative-multi-agent-q-learning-with","repo_url":"https://github.com/opendilab/ace","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"multi-agent-reinforcement-learning","task_name":"Multi-agent Reinforcement Learning"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"smac","task_name":"SMAC"},{"task_slug":"smac-1","task_name":"SMAC+"},{"task_slug":"sequential-decision-making","task_name":"Sequential Decision Making"},{"task_slug":"starcraft","task_name":"Starcraft"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"biggan","method_name":"BigGAN"},{"method_slug":"conditional-batch-normalization","method_name":"Conditional Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"early-stopping","method_name":"Early Stopping"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"gan-hinge-loss","method_name":"GAN Hinge Loss"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"non-local-block","method_name":"Non-Local Block"},{"method_slug":"non-local-operation","method_name":"Non-Local Operation"},{"method_slug":"off-diagonal-orthogonal-regularization","method_name":"Off-Diagonal Orthogonal Regularization"},{"method_slug":"projection-discriminator","method_name":"Projection Discriminator"},{"method_slug":"q-learning","method_name":"Q-Learning"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sagan","method_name":"SAGAN"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"spectral-normalization","method_name":"Spectral Normalization"},{"method_slug":"ttur","method_name":"TTUR"},{"method_slug":"truncation-trick","method_name":"Truncation Trick"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/smac-on-smac-3s5z-vs-3s6z-1","task":"SMAC","dataset":"SMAC 3s5z_vs_3s6z","model":"ACE","rank_in_archive_order":1,"of":13,"metrics":{"Median Win Rate":"100"},"uses_additional_data":false},{"leaderboard":"/sota/smac-on-smac-6h-vs-8z-1","task":"SMAC","dataset":"SMAC 6h_vs_8z","model":"ACE","rank_in_archive_order":1,"of":14,"metrics":{"Median Win Rate":"93.75"},"uses_additional_data":false},{"leaderboard":"/sota/smac-on-smac-mmm2-1","task":"SMAC","dataset":"SMAC MMM2","model":"ACE","rank_in_archive_order":1,"of":14,"metrics":{"Median Win Rate":"100"},"uses_additional_data":false},{"leaderboard":"/sota/smac-on-smac-corridor","task":"SMAC","dataset":"SMAC corridor","model":"ACE","rank_in_archive_order":1,"of":13,"metrics":{"Median Win Rate":"100"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.16068","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.16068"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opendilab/ace","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ab128bc8ab5f6d4a","entry":"ActionSampler","repo":"opendilab/ace","repo_kind":"official","path":"ding/model/template/grf_ace.py","file_url":"https://github.com/opendilab/ace/blob/HEAD/ding/model/template/grf_ace.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab128bc8ab5f6d4a"}},{"code_sha256_prefix":"e1b90070a1e51736","entry":"concat_state_action_pairs","repo":"opendilab/ace","repo_kind":"official","path":"ding/reward_model/gail_irl_model.py","file_url":"https://github.com/opendilab/ace/blob/HEAD/ding/reward_model/gail_irl_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1b90070a1e51736"}},{"code_sha256_prefix":"a074649f8b341a8a","entry":"pfsp","repo":"opendilab/ace","repo_kind":"official","path":"ding/league/algorithm.py","file_url":"https://github.com/opendilab/ace/blob/HEAD/ding/league/algorithm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a074649f8b341a8a"}},{"code_sha256_prefix":"c7dde4c97b7136b3","entry":"uniform","repo":"opendilab/ace","repo_kind":"official","path":"ding/league/algorithm.py","file_url":"https://github.com/opendilab/ace/blob/HEAD/ding/league/algorithm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7dde4c97b7136b3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}