{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethink-global-reward-game-and-credit","title":"Shapley Q-value: A Local Reward Approach to Solve Global Reward Games","arxiv_id":"1907.05707","date":"2019-07-11","proceeding":null,"authors":["Jianhong Wang","Yuan Zhang","Tae-Kyun Kim","Yunjie Gu"],"abstract":"Cooperative game is a critical research area in the multi-agent reinforcement learning (MARL). Global reward game is a subclass of cooperative games, where all agents aim to maximize the global reward. Credit assignment is an important problem studied in the global reward game. Most of previous works stood by the view of non-cooperative-game theoretical framework with the shared reward approach, i.e., each agent being assigned a shared global reward directly. This, however, may give each agent an inaccurate reward on its contribution to the group, which could cause inefficient learning. To deal with this problem, we i) introduce a cooperative-game theoretical framework called extended convex game (ECG) that is a superset of global reward game, and ii) propose a local reward approach called Shapley Q-value. Shapley Q-value is able to distribute the global reward, reflecting each agent's own contribution in contrast to the shared reward approach. Moreover, we derive an MARL algorithm called Shapley Q-value deep deterministic policy gradient (SQDDPG), using Shapley Q-value as the critic for each agent. We evaluate SQDDPG on Cooperative Navigation, Prey-and-Predator and Traffic Junction, compared with the state-of-the-art algorithms, e.g., MADDPG, COMA, Independent DDPG and Independent A2C. In the experiments, SQDDPG shows a significant improvement on the convergence rate. Finally, we plot Shapley Q-value and validate the property of fair credit assignment.","url_abs":"https://arxiv.org/abs/1907.05707v6","url_pdf":"https://arxiv.org/pdf/1907.05707v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethink-global-reward-game-and-credit","repo_url":"https://github.com/hsvgbkhgbv/SQDDPG","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"rethink-global-reward-game-and-credit","repo_url":"https://github.com/Multi-Agent-Research-Group/SQDDPG","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"multi-agent-reinforcement-learning","task_name":"Multi-agent Reinforcement Learning"},{"task_slug":"policy-gradient-methods","task_name":"Policy Gradient Methods"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"ddpg","method_name":"DDPG"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"experience-replay","method_name":"Experience Replay"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1907.05707","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.05707"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Multi-Agent-Research-Group/SQDDPG","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hsvgbkhgbv/SQDDPG","reach":null}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"617ff2b6609d7576","entry":"Model","repo":"hsvgbkhgbv/SQDDPG","repo_kind":"official","path":"models/sqddpg.py","file_url":"https://github.com/hsvgbkhgbv/SQDDPG/blob/HEAD/models/sqddpg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"617ff2b6609d7576"}},{"code_sha256_prefix":"c8c7bd5b8676a2ce","entry":"SQDDPG","repo":"hsvgbkhgbv/SQDDPG","repo_kind":"official","path":"models/sqddpg.py","file_url":"https://github.com/hsvgbkhgbv/SQDDPG/blob/HEAD/models/sqddpg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8c7bd5b8676a2ce"}},{"code_sha256_prefix":"0fb15e17be7b4141","entry":"SQDDPG","repo":"Multi-Agent-Research-Group/SQDDPG","repo_kind":"listed","path":"models/sqddpg.py","file_url":"https://github.com/Multi-Agent-Research-Group/SQDDPG/blob/HEAD/models/sqddpg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fb15e17be7b4141"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}