{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/solving-common-payoff-games-with-approximate","title":"Solving Common-Payoff Games with Approximate Policy Iteration","arxiv_id":"2101.04237","date":"2021-01-11","proceeding":null,"authors":["Samuel Sokota","Edward Lockhart","Finbarr Timbers","Elnaz Davoodi","Ryan D'Orazio","Neil Burch","Martin Schmid","Michael Bowling","Marc Lanctot"],"abstract":"For artificially intelligent learning systems to have widespread applicability in real-world settings, it is important that they be able to operate decentrally. Unfortunately, decentralized control is difficult -- computing even an epsilon-optimal joint policy is a NEXP complete problem. Nevertheless, a recently rediscovered insight -- that a team of agents can coordinate via common knowledge -- has given rise to algorithms capable of finding optimal joint policies in small common-payoff games. The Bayesian action decoder (BAD) leverages this insight and deep reinforcement learning to scale to games as large as two-player Hanabi. However, the approximations it uses to do so prevent it from discovering optimal joint policies even in games small enough to brute force optimal solutions. This work proposes CAPI, a novel algorithm which, like BAD, combines common knowledge with deep reinforcement learning. However, unlike BAD, CAPI prioritizes the propensity to discover optimal joint policies over scalability. While this choice precludes CAPI from scaling to games as large as Hanabi, empirical results demonstrate that, on the games to which CAPI does scale, it is capable of discovering optimal joint policies even when other modern multi-agent reinforcement learning algorithms are unable to do so. Code is available at https://github.com/ssokota/capi .","url_abs":"https://arxiv.org/abs/2101.04237v1","url_pdf":"https://arxiv.org/pdf/2101.04237v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"solving-common-payoff-games-with-approximate","repo_url":"https://github.com/ssokota/capi","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"solving-common-payoff-games-with-approximate","repo_url":"https://github.com/ssokota/tiny-hanabi","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"multi-agent-reinforcement-learning","task_name":"Multi-agent Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2101.04237","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2101.04237"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ssokota/capi","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/ssokota/tiny-hanabi","reach":{"status":"unanswered"}}],"summary":{"ran":2,"unverified":7},"by_repo_kind":{"official":{"samples":9,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4d846c442d1f74cd","entry":"NN","repo":"ssokota/capi","repo_kind":"official","path":"capi/agents.py","file_url":"https://github.com/ssokota/capi/blob/HEAD/capi/agents.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d846c442d1f74cd"}},{"code_sha256_prefix":"6a41dfa5b1829b26","entry":"State","repo":"ssokota/capi","repo_kind":"official","path":"capi/agents.py","file_url":"https://github.com/ssokota/capi/blob/HEAD/capi/agents.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a41dfa5b1829b26"}},{"code_sha256_prefix":"f75ce66758a1ed20","entry":"Agent","repo":"ssokota/capi","repo_kind":"official","path":"capi/agents.py","file_url":"https://github.com/ssokota/capi/blob/HEAD/capi/agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f75ce66758a1ed20"}},{"code_sha256_prefix":"f64a58fc81c08af4","entry":"assert_element","repo":"ssokota/capi","repo_kind":"official","path":"capi/agents.py","file_url":"https://github.com/ssokota/capi/blob/HEAD/capi/agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f64a58fc81c08af4"}},{"code_sha256_prefix":"625d6e259bb25eed","entry":"assert_joint_probability","repo":"ssokota/capi","repo_kind":"official","path":"capi/agents.py","file_url":"https://github.com/ssokota/capi/blob/HEAD/capi/agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"625d6e259bb25eed"}},{"code_sha256_prefix":"bfb9f5a5d8c5237f","entry":"assert_label_prescription","repo":"ssokota/capi","repo_kind":"official","path":"capi/agents.py","file_url":"https://github.com/ssokota/capi/blob/HEAD/capi/agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bfb9f5a5d8c5237f"}},{"code_sha256_prefix":"1f06f9d6110f4d9e","entry":"assert_num_dims","repo":"ssokota/capi","repo_kind":"official","path":"capi/agents.py","file_url":"https://github.com/ssokota/capi/blob/HEAD/capi/agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f06f9d6110f4d9e"}},{"code_sha256_prefix":"8aedc01b97e4426d","entry":"assert_prescription","repo":"ssokota/capi","repo_kind":"official","path":"capi/agents.py","file_url":"https://github.com/ssokota/capi/blob/HEAD/capi/agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8aedc01b97e4426d"}},{"code_sha256_prefix":"6ed2f3f73ed808fe","entry":"assert_shape","repo":"ssokota/capi","repo_kind":"official","path":"capi/agents.py","file_url":"https://github.com/ssokota/capi/blob/HEAD/capi/agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ed2f3f73ed808fe"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}