{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/smoa-improving-multi-agent-large-language","title":"SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents","arxiv_id":"2411.03284","date":"2024-11-05","proceeding":null,"authors":["Dawei Li","Zhen Tan","Peijia Qian","YiFan Li","Kumar Satvik Chaudhary","Lijie Hu","Jiayi Shen"],"abstract":"While multi-agent systems have been shown to significantly enhance the performance of Large Language Models (LLMs) across various tasks and applications, the dense interaction between scaling agents potentially hampers their efficiency and diversity. To address these challenges, we draw inspiration from the sparse mixture-of-agents (SMoE) and propose a sparse mixture-of-agents (SMoA) framework to improve the efficiency and diversity of multi-agent LLMs. Unlike completely connected structures, SMoA introduces novel Response Selection and Early Stopping mechanisms to sparsify information flows among individual LLM agents, striking a balance between performance and efficiency. Additionally, inspired by the expert diversity principle in SMoE frameworks for workload balance between experts, we assign distinct role descriptions to each LLM agent, fostering diverse and divergent thinking. Extensive experiments on reasoning, alignment, and fairness benchmarks demonstrate that SMoA achieves performance comparable to traditional mixture-of-agents approaches but with significantly lower computational costs. Further analysis reveals that SMoA is more stable, has a greater capacity to scale, and offers considerable potential through hyper-parameter optimization. Code and data will be available at: https://github.com/David-Li0406/SMoA.","url_abs":"https://arxiv.org/abs/2411.03284v1","url_pdf":"https://arxiv.org/pdf/2411.03284v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"smoa-improving-multi-agent-large-language","repo_url":"https://github.com/david-li0406/smoa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"fairness","task_name":"Fairness"}],"methods":[{"method_slug":"early-stopping","method_name":"Early Stopping"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.03284","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.03284"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/david-li0406/smoa","reach":{"status":"ok"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"c4be2ee7c659a23f","entry":"call_api","repo":"david-li0406/smoa","repo_kind":"official","path":"eval_ceb.py","file_url":"https://github.com/david-li0406/smoa/blob/HEAD/eval_ceb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4be2ee7c659a23f"}},{"code_sha256_prefix":"344d6a8a83ac7a6e","entry":"convert_to_string","repo":"david-li0406/smoa","repo_kind":"official","path":"run_mmau.py","file_url":"https://github.com/david-li0406/smoa/blob/HEAD/run_mmau.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"344d6a8a83ac7a6e"}},{"code_sha256_prefix":"b1081ff75f6e7854","entry":"generate_openai","repo":"david-li0406/smoa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/david-li0406/smoa/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1081ff75f6e7854"}},{"code_sha256_prefix":"3b825d4c554f398b","entry":"generate_together","repo":"david-li0406/smoa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/david-li0406/smoa/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b825d4c554f398b"}},{"code_sha256_prefix":"af39cecf797861bd","entry":"generate_together_stream","repo":"david-li0406/smoa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/david-li0406/smoa/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af39cecf797861bd"}},{"code_sha256_prefix":"9f11a99fca0a359b","entry":"process_fn","repo":"david-li0406/smoa","repo_kind":"official","path":"run_alignment.py","file_url":"https://github.com/david-li0406/smoa/blob/HEAD/run_alignment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f11a99fca0a359b"}},{"code_sha256_prefix":"cecfd787af984f73","entry":"process_fn","repo":"david-li0406/smoa","repo_kind":"official","path":"run_ceb.py","file_url":"https://github.com/david-li0406/smoa/blob/HEAD/run_ceb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cecfd787af984f73"}},{"code_sha256_prefix":"e6a8765061e98ad2","entry":"process_fn","repo":"david-li0406/smoa","repo_kind":"official","path":"run_mmau.py","file_url":"https://github.com/david-li0406/smoa/blob/HEAD/run_mmau.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6a8765061e98ad2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}