{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/influencing-long-term-behavior-in-multiagent","title":"Influencing Long-Term Behavior in Multiagent Reinforcement Learning","arxiv_id":"2203.03535","date":"2022-03-07","proceeding":null,"authors":["Dong-Ki Kim","Matthew Riemer","Miao Liu","Jakob N. Foerster","Michael Everett","Chuangchuang Sun","Gerald Tesauro","Jonathan P. How"],"abstract":"The main challenge of multiagent reinforcement learning is the difficulty of learning useful policies in the presence of other simultaneously learning agents whose changing behaviors jointly affect the environment's transition and reward dynamics. An effective approach that has recently emerged for addressing this non-stationarity is for each agent to anticipate the learning of other agents and influence the evolution of future policies towards desirable behavior for its own benefit. Unfortunately, previous approaches for achieving this suffer from myopic evaluation, considering only a finite number of policy updates. As such, these methods can only influence transient future policies rather than achieving the promise of scalable equilibrium selection approaches that influence the behavior at convergence. In this paper, we propose a principled framework for considering the limiting policies of other agents as time approaches infinity. Specifically, we develop a new optimization objective that maximizes each agent's average reward by directly accounting for the impact of its behavior on the limiting set of policies that other agents will converge to. Our paper characterizes desirable solution concepts within this problem setting and provides practical approaches for optimizing over possible outcomes. As a result of our farsighted objective, we demonstrate better long-term performance than state-of-the-art baselines across a suite of diverse multiagent benchmark domains.","url_abs":"https://arxiv.org/abs/2203.03535v4","url_pdf":"https://arxiv.org/pdf/2203.03535v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"influencing-long-term-behavior-in-multiagent","repo_url":"https://github.com/dkkim93/further","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.03535","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.03535"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dkkim93/further","reach":null}],"summary":{"ran":3,"ran_draft_wrong":5,"ran_honours":1,"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e953502a76ebafce","entry":"CategoricalMLP","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e953502a76ebafce"}},{"code_sha256_prefix":"e2120bd5095898b6","entry":"GaussianMLP","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2120bd5095898b6"}},{"code_sha256_prefix":"6cc0afec8e5e5562","entry":"ReplayMemory","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6cc0afec8e5e5562"}},{"code_sha256_prefix":"5d15c5576b6f1119","entry":"get_env_action_type","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d15c5576b6f1119"}},{"code_sha256_prefix":"3cc888ba40ec94e8","entry":"get_env_dim","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3cc888ba40ec94e8"}},{"code_sha256_prefix":"a460d87b686cc889","entry":"get_parameters","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a460d87b686cc889"}},{"code_sha256_prefix":"92c1700ad0ee80cf","entry":"reparameterization","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92c1700ad0ee80cf"}},{"code_sha256_prefix":"4f00b7f646079156","entry":"to_numpy","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f00b7f646079156"}},{"code_sha256_prefix":"818ef30cb3e0a466","entry":"to_onehot","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"818ef30cb3e0a466"}},{"code_sha256_prefix":"9b6789716d0eab8d","entry":"to_transition","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b6789716d0eab8d"}},{"code_sha256_prefix":"992fa4129e010d32","entry":"Base","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"992fa4129e010d32"}},{"code_sha256_prefix":"662aff050d002949","entry":"FURTHER","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"662aff050d002949"}},{"code_sha256_prefix":"cb1c431c7a635b3f","entry":"weight_init","repo":"dkkim93/further","repo_kind":"official","path":"algorithm/further/agent.py","file_url":"https://github.com/dkkim93/further/blob/HEAD/algorithm/further/agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb1c431c7a635b3f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}