{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/shapley-machine-a-game-theoretic-framework","title":"Shapley Machine: A Game-Theoretic Framework for N-Agent Ad Hoc Teamwork","arxiv_id":"2506.11285","date":"2025-06-12","proceeding":null,"authors":["Jianhong Wang","Yang Li","Samuel Kaski","Jonathan Lawry"],"abstract":"Open multi-agent systems are increasingly important in modeling real-world applications, such as smart grids, swarm robotics, etc. In this paper, we aim to investigate a recently proposed problem for open multi-agent systems, referred to as n-agent ad hoc teamwork (NAHT), where only a number of agents are controlled. Existing methods tend to be based on heuristic design and consequently lack theoretical rigor and ambiguous credit assignment among agents. To address these limitations, we model and solve NAHT through the lens of cooperative game theory. More specifically, we first model an open multi-agent system, characterized by its value, as an instance situated in a space of cooperative games, generated by a set of basis games. We then extend this space, along with the state space, to accommodate dynamic scenarios, thereby characterizing NAHT. Exploiting the justifiable assumption that basis game values correspond to a sequence of n-step returns with different horizons, we represent the state values for NAHT in a form similar to $\\lambda$-returns. Furthermore, we derive Shapley values to allocate state values to the controlled agents, as credits for their contributions to the ad hoc team. Different from the conventional approach to shaping Shapley values in an explicit form, we shape Shapley values by fulfilling the three axioms uniquely describing them, well defined on the extended game space describing NAHT. To estimate Shapley values in dynamic scenarios, we propose a TD($\\lambda$)-like algorithm. The resulting reinforcement learning (RL) algorithm is referred to as Shapley Machine. To our best knowledge, this is the first time that the concepts from cooperative game theory are directly related to RL concepts. In experiments, we demonstrate the effectiveness of Shapley Machine and verify reasonableness of our theory.","url_abs":"https://arxiv.org/abs/2506.11285v1","url_pdf":"https://arxiv.org/pdf/2506.11285v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"shapley-machine-a-game-theoretic-framework","repo_url":"https://github.com/hsvgbkhgbv/shapley-machine-naht","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"hoc","method_name":"HOC"},{"method_slug":"n-step-returns","method_name":"N-step Returns"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.11285","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.11285"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hsvgbkhgbv/shapley-machine-naht","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"355d7b51c7ffd042","entry":"gumbel_softmax_sample","repo":"hsvgbkhgbv/shapley-machine-naht","repo_kind":"official","path":"src/controllers/maddpg_controller.py","file_url":"https://github.com/hsvgbkhgbv/shapley-machine-naht/blob/HEAD/src/controllers/maddpg_controller.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"355d7b51c7ffd042"}},{"code_sha256_prefix":"046b92243aae4f76","entry":"onehot_from_logits","repo":"hsvgbkhgbv/shapley-machine-naht","repo_kind":"official","path":"src/controllers/maddpg_controller.py","file_url":"https://github.com/hsvgbkhgbv/shapley-machine-naht/blob/HEAD/src/controllers/maddpg_controller.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"046b92243aae4f76"}},{"code_sha256_prefix":"3695d66ab681cacf","entry":"sample_gumbel","repo":"hsvgbkhgbv/shapley-machine-naht","repo_kind":"official","path":"src/controllers/maddpg_controller.py","file_url":"https://github.com/hsvgbkhgbv/shapley-machine-naht/blob/HEAD/src/controllers/maddpg_controller.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3695d66ab681cacf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}