{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kaleidoscope-learnable-masks-for","title":"Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learning","arxiv_id":"2410.08540","date":"2024-10-11","proceeding":null,"authors":["Xinran Li","Ling Pan","Jun Zhang"],"abstract":"In multi-agent reinforcement learning (MARL), parameter sharing is commonly employed to enhance sample efficiency. However, the popular approach of full parameter sharing often leads to homogeneous policies among agents, potentially limiting the performance benefits that could be derived from policy diversity. To address this critical limitation, we introduce \\emph{Kaleidoscope}, a novel adaptive partial parameter sharing scheme that fosters policy heterogeneity while still maintaining high sample efficiency. Specifically, Kaleidoscope maintains one set of common parameters alongside multiple sets of distinct, learnable masks for different agents, dictating the sharing of parameters. It promotes diversity among policy networks by encouraging discrepancy among these masks, without sacrificing the efficiencies of parameter sharing. This design allows Kaleidoscope to dynamically balance high sample efficiency with a broad policy representational capacity, effectively bridging the gap between full parameter sharing and non-parameter sharing across various environments. We further extend Kaleidoscope to critic ensembles in the context of actor-critic algorithms, which could help improve value estimations.Our empirical evaluations across extensive environments, including multi-agent particle environment, multi-agent MuJoCo and StarCraft multi-agent challenge v2, demonstrate the superior performance of Kaleidoscope compared with existing parameter sharing approaches, showcasing its potential for performance enhancement in MARL. The code is publicly available at \\url{https://github.com/LXXXXR/Kaleidoscope}.","url_abs":"https://arxiv.org/abs/2410.08540v1","url_pdf":"https://arxiv.org/pdf/2410.08540v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kaleidoscope-learnable-masks-for","repo_url":"https://github.com/lxxxxr/kaleidoscope","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"multi-agent-reinforcement-learning","task_name":"Multi-agent Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"starcraft","task_name":"Starcraft"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.08540","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.08540"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LXXXXR/Kaleidoscope","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lxxxxr/kaleidoscope","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"ran_draft_wrong":3,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1},"community":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3e4047b55fdd8d46","entry":"KaleiLinear","repo":"lxxxxr/kaleidoscope","repo_kind":"official","path":"Kalei_MPE/src/modules/agents/rnn_agent.py","file_url":"https://github.com/lxxxxr/kaleidoscope/blob/HEAD/Kalei_MPE/src/modules/agents/rnn_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e4047b55fdd8d46"}},{"code_sha256_prefix":"d10c169fc33ec23c","entry":"compare_STE","repo":"lxxxxr/kaleidoscope","repo_kind":"official","path":"Kalei_MPE/src/modules/agents/rnn_agent.py","file_url":"https://github.com/lxxxxr/kaleidoscope/blob/HEAD/Kalei_MPE/src/modules/agents/rnn_agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d10c169fc33ec23c"}},{"code_sha256_prefix":"868161eada09b424","entry":"extract_alg_name_from_config","repo":"uoe-agents/epymarl","repo_kind":"community","path":"plot_results.py","file_url":"https://github.com/uoe-agents/epymarl/blob/HEAD/plot_results.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"868161eada09b424"}},{"code_sha256_prefix":"b23d20b93b147852","entry":"extract_env_name_from_config","repo":"uoe-agents/epymarl","repo_kind":"community","path":"plot_results.py","file_url":"https://github.com/uoe-agents/epymarl/blob/HEAD/plot_results.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b23d20b93b147852"}},{"code_sha256_prefix":"35b9956a050d423d","entry":"flatdim_box_multibinary","repo":"LXXXXR/Kaleidoscope","repo_kind":"official","path":"Kalei_MPE/src/envs/gym_spaces_utils.py","file_url":"https://github.com/LXXXXR/Kaleidoscope/blob/HEAD/Kalei_MPE/src/envs/gym_spaces_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"35b9956a050d423d"}},{"code_sha256_prefix":"33d389e8d257639f","entry":"flatdim_discrete","repo":"LXXXXR/Kaleidoscope","repo_kind":"official","path":"Kalei_MPE/src/envs/gym_spaces_utils.py","file_url":"https://github.com/LXXXXR/Kaleidoscope/blob/HEAD/Kalei_MPE/src/envs/gym_spaces_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"33d389e8d257639f"}},{"code_sha256_prefix":"355d7b51c7ffd042","entry":"gumbel_softmax_sample","repo":"LXXXXR/Kaleidoscope","repo_kind":"official","path":"Kalei_MPE/src/controllers/maddpg_controller.py","file_url":"https://github.com/LXXXXR/Kaleidoscope/blob/HEAD/Kalei_MPE/src/controllers/maddpg_controller.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"355d7b51c7ffd042"}},{"code_sha256_prefix":"e8a91a919b93f3e1","entry":"load_results","repo":"uoe-agents/epymarl","repo_kind":"community","path":"plot_results.py","file_url":"https://github.com/uoe-agents/epymarl/blob/HEAD/plot_results.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8a91a919b93f3e1"}},{"code_sha256_prefix":"046b92243aae4f76","entry":"onehot_from_logits","repo":"LXXXXR/Kaleidoscope","repo_kind":"official","path":"Kalei_MPE/src/controllers/maddpg_controller.py","file_url":"https://github.com/LXXXXR/Kaleidoscope/blob/HEAD/Kalei_MPE/src/controllers/maddpg_controller.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"046b92243aae4f76"}},{"code_sha256_prefix":"3695d66ab681cacf","entry":"sample_gumbel","repo":"LXXXXR/Kaleidoscope","repo_kind":"official","path":"Kalei_MPE/src/controllers/maddpg_controller.py","file_url":"https://github.com/LXXXXR/Kaleidoscope/blob/HEAD/Kalei_MPE/src/controllers/maddpg_controller.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3695d66ab681cacf"}},{"code_sha256_prefix":"930812c3b06fc858","entry":"work","repo":"LXXXXR/Kaleidoscope","repo_kind":"official","path":"Kalei_MPE/src/search.py","file_url":"https://github.com/LXXXXR/Kaleidoscope/blob/HEAD/Kalei_MPE/src/search.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"930812c3b06fc858"}},{"code_sha256_prefix":"73ec7f831d9eb7bc","entry":"Kalei_RNNAgent_1R3","repo":"lxxxxr/kaleidoscope","repo_kind":"official","path":"Kalei_MPE/src/modules/agents/rnn_agent.py","file_url":"https://github.com/lxxxxr/kaleidoscope/blob/HEAD/Kalei_MPE/src/modules/agents/rnn_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73ec7f831d9eb7bc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}