{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/knowing-what-not-to-do-leverage-language","title":"Knowing What Not to Do: Leverage Language Model Insights for Action Space Pruning in Multi-agent Reinforcement Learning","arxiv_id":"2405.16854","date":"2024-05-27","proceeding":null,"authors":["Zhihao Liu","Xianliang Yang","Zichuan Liu","Yifan Xia","Wei Jiang","Yuanyu Zhang","Lijuan Li","Guoliang Fan","Lei Song","Bian Jiang"],"abstract":"Multi-agent reinforcement learning (MARL) is employed to develop autonomous agents that can learn to adopt cooperative or competitive strategies within complex environments. However, the linear increase in the number of agents leads to a combinatorial explosion of the action space, which may result in algorithmic instability, difficulty in convergence, or entrapment in local optima. While researchers have designed a variety of effective algorithms to compress the action space, these methods also introduce new challenges, such as the need for manually designed prior knowledge or reliance on the structure of the problem, which diminishes the applicability of these techniques. In this paper, we introduce Evolutionary action SPAce Reduction with Knowledge (eSpark), an exploration function generation framework driven by large language models (LLMs) to boost exploration and prune unnecessary actions in MARL. Using just a basic prompt that outlines the overall task and setting, eSpark is capable of generating exploration functions in a zero-shot manner, identifying and pruning redundant or irrelevant state-action pairs, and then achieving autonomous improvement from policy feedback. In reinforcement learning tasks involving inventory management and traffic light control encompassing a total of 15 scenarios, eSpark consistently outperforms the combined MARL algorithm in all scenarios, achieving an average performance gain of 34.4% and 9.9% in the two types of tasks respectively. Additionally, eSpark has proven to be capable of managing situations with a large number of agents, securing a 29.7% improvement in scalability challenges that featured over 500 agents. The code can be found in https://github.com/LiuZhihao2022/eSpark.git.","url_abs":"https://arxiv.org/abs/2405.16854v1","url_pdf":"https://arxiv.org/pdf/2405.16854v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"knowing-what-not-to-do-leverage-language","repo_url":"https://github.com/liuzhihao2022/espark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.16854","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.16854"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liuzhihao2022/espark","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":10,"unverified":1},"by_repo_kind":{"official":{"samples":11,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"19b86caa10272f9b","entry":"assign_gpu","repo":"liuzhihao2022/espark","repo_kind":"official","path":"Baseline/MARL_algorithm/run/eSpark_no_llm_run.py","file_url":"https://github.com/liuzhihao2022/espark/blob/HEAD/Baseline/MARL_algorithm/run/eSpark_no_llm_run.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"19b86caa10272f9b"}},{"code_sha256_prefix":"6008b0eb92d8067c","entry":"categorical_entropy","repo":"liuzhihao2022/espark","repo_kind":"official","path":"Baseline/MARL_algorithm/components/action_selectors.py","file_url":"https://github.com/liuzhihao2022/espark/blob/HEAD/Baseline/MARL_algorithm/components/action_selectors.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6008b0eb92d8067c"}},{"code_sha256_prefix":"883ff631b5f026bc","entry":"divide_arr_into_interval","repo":"liuzhihao2022/espark","repo_kind":"official","path":"Baseline/MARL_algorithm/run/eSpark_no_llm_run.py","file_url":"https://github.com/liuzhihao2022/espark/blob/HEAD/Baseline/MARL_algorithm/run/eSpark_no_llm_run.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"883ff631b5f026bc"}},{"code_sha256_prefix":"5c2cd0041222a356","entry":"get_multilevel_single_stock_level","repo":"liuzhihao2022/espark","repo_kind":"official","path":"Baseline/OR_algorithm/base_stock.py","file_url":"https://github.com/liuzhihao2022/espark/blob/HEAD/Baseline/OR_algorithm/base_stock.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c2cd0041222a356"}},{"code_sha256_prefix":"ea5ed37f3e200dc7","entry":"get_multilevel_stock_level","repo":"liuzhihao2022/espark","repo_kind":"official","path":"Baseline/OR_algorithm/base_stock.py","file_url":"https://github.com/liuzhihao2022/espark/blob/HEAD/Baseline/OR_algorithm/base_stock.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ea5ed37f3e200dc7"}},{"code_sha256_prefix":"1e2aef5f1efa0988","entry":"get_ranks","repo":"liuzhihao2022/espark","repo_kind":"official","path":"Baseline/OR_algorithm/search_different_profit.py","file_url":"https://github.com/liuzhihao2022/espark/blob/HEAD/Baseline/OR_algorithm/search_different_profit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e2aef5f1efa0988"}},{"code_sha256_prefix":"4164e21ded1cfc42","entry":"multilevel_base_stock","repo":"liuzhihao2022/espark","repo_kind":"official","path":"Baseline/OR_algorithm/base_stock.py","file_url":"https://github.com/liuzhihao2022/espark/blob/HEAD/Baseline/OR_algorithm/base_stock.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4164e21ded1cfc42"}},{"code_sha256_prefix":"2acbf488aab98339","entry":"multinomial_entropy","repo":"liuzhihao2022/espark","repo_kind":"official","path":"Baseline/MARL_algorithm/components/action_selectors.py","file_url":"https://github.com/liuzhihao2022/espark/blob/HEAD/Baseline/MARL_algorithm/components/action_selectors.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2acbf488aab98339"}},{"code_sha256_prefix":"8602497bc824a734","entry":"sS_policy","repo":"liuzhihao2022/espark","repo_kind":"official","path":"Baseline/OR_algorithm/search_sS.py","file_url":"https://github.com/liuzhihao2022/espark/blob/HEAD/Baseline/OR_algorithm/search_sS.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8602497bc824a734"}},{"code_sha256_prefix":"530140af058c05b0","entry":"search_sS","repo":"liuzhihao2022/espark","repo_kind":"official","path":"Baseline/OR_algorithm/search_sS.py","file_url":"https://github.com/liuzhihao2022/espark/blob/HEAD/Baseline/OR_algorithm/search_sS.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"530140af058c05b0"}},{"code_sha256_prefix":"bac932c18ae61250","entry":"get_ranks","repo":"liuzhihao2022/espark","repo_kind":"official","path":"Baseline/OR_algorithm/random_actions.py","file_url":"https://github.com/liuzhihao2022/espark/blob/HEAD/Baseline/OR_algorithm/random_actions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bac932c18ae61250"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}