{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/alberdice-addressing-out-of-distribution-1","title":"AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation","arxiv_id":"2311.02194","date":"2023-11-03","proceeding":"NeurIPS 2023 11","authors":["Daiki E. Matsunaga","Jongmin Lee","Jaeseok Yoon","Stefanos Leonardos","Pieter Abbeel","Kee-Eung Kim"],"abstract":"One of the main challenges in offline Reinforcement Learning (RL) is the distribution shift that arises from the learned policy deviating from the data collection policy. This is often addressed by avoiding out-of-distribution (OOD) actions during policy improvement as their presence can lead to substantial performance degradation. This challenge is amplified in the offline Multi-Agent RL (MARL) setting since the joint action space grows exponentially with the number of agents. To avoid this curse of dimensionality, existing MARL methods adopt either value decomposition methods or fully decentralized training of individual agents. However, even when combined with standard conservatism principles, these methods can still result in the selection of OOD joint actions in offline MARL. To this end, we introduce AlberDICE, an offline MARL algorithm that alternatively performs centralized training of individual agents based on stationary distribution optimization. AlberDICE circumvents the exponential complexity of MARL by computing the best response of one agent at a time while effectively avoiding OOD joint action selection. Theoretically, we show that the alternating optimization procedure converges to Nash policies. In the experiments, we demonstrate that AlberDICE significantly outperforms baseline algorithms on a standard suite of MARL benchmarks.","url_abs":"https://arxiv.org/abs/2311.02194v1","url_pdf":"https://arxiv.org/pdf/2311.02194v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"alberdice-addressing-out-of-distribution-1","repo_url":"https://github.com/dematsunaga/alberdice","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.02194","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.02194"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dematsunaga/alberdice","reach":null}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"a3eae393addab202","entry":"EpisodeBatch","repo":"dematsunaga/alberdice","repo_kind":"official","path":"pymarl-master/src/learners/alberdice_learner.py","file_url":"https://github.com/dematsunaga/alberdice/blob/HEAD/pymarl-master/src/learners/alberdice_learner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a3eae393addab202"}},{"code_sha256_prefix":"a13a12eedfd267c7","entry":"OptiDiceCritic","repo":"dematsunaga/alberdice","repo_kind":"official","path":"pymarl-master/src/learners/alberdice_learner.py","file_url":"https://github.com/dematsunaga/alberdice/blob/HEAD/pymarl-master/src/learners/alberdice_learner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a13a12eedfd267c7"}},{"code_sha256_prefix":"fabab1ad75467062","entry":"AlberDiceLearner","repo":"dematsunaga/alberdice","repo_kind":"official","path":"pymarl-master/src/learners/alberdice_learner.py","file_url":"https://github.com/dematsunaga/alberdice/blob/HEAD/pymarl-master/src/learners/alberdice_learner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fabab1ad75467062"}},{"code_sha256_prefix":"b1c71a9f4ec4e185","entry":"BasicMAC","repo":"dematsunaga/alberdice","repo_kind":"official","path":"pymarl-master/src/learners/alberdice_learner.py","file_url":"https://github.com/dematsunaga/alberdice/blob/HEAD/pymarl-master/src/learners/alberdice_learner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1c71a9f4ec4e185"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}