{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mapcoder-multi-agent-code-generation-for","title":"MapCoder: Multi-Agent Code Generation for Competitive Problem Solving","arxiv_id":"2405.11403","date":"2024-05-18","proceeding":null,"authors":["Md. Ashraful Islam","Mohammed Eunus Ali","Md Rizwan Parvez"],"abstract":"Code synthesis, which requires a deep understanding of complex natural language problem descriptions, generation of code instructions for complex algorithms and data structures, and the successful execution of comprehensive unit tests, presents a significant challenge. While large language models (LLMs) demonstrate impressive proficiency in natural language processing, their performance in code generation tasks remains limited. In this paper, we introduce a new approach to code generation tasks leveraging multi-agent prompting that uniquely replicates the full cycle of program synthesis as observed in human developers. Our framework, MapCoder, consists of four LLM agents specifically designed to emulate the stages of this cycle: recalling relevant examples, planning, code generation, and debugging. After conducting thorough experiments, with multiple LLM ablations and analyses across eight challenging competitive problem-solving and program synthesis benchmarks, MapCoder showcases remarkable code generation capabilities, achieving new state-of-the-art results (pass@1) on HumanEval (93.9%), MBPP (83.1%), APPS (22.0%), CodeContests (28.5%), and xCodeEval (45.3%). Moreover, our method consistently delivers superior performance across various programming languages and varying problem difficulties. We open-source our framework at https://github.com/Md-Ashraful-Pramanik/MapCoder.","url_abs":"https://arxiv.org/abs/2405.11403v1","url_pdf":"https://arxiv.org/pdf/2405.11403v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mapcoder-multi-agent-code-generation-for","repo_url":"https://github.com/md-ashraful-pramanik/mapcoder","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"mapcoder-multi-agent-code-generation-for","repo_url":"https://github.com/Luoji-zju/Agents4PLC_release","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"humaneval","task_name":"HumanEval"},{"task_slug":"program-synthesis","task_name":"Program Synthesis"},{"task_slug":null,"task_name":"mbpp"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/code-generation-on-apps","task":"Code Generation","dataset":"APPS","model":"MapCoder APPS-150-cherrypicked (GPT-4)","rank_in_archive_order":16,"of":18,"metrics":{"Competition Pass@1":"0.00%","Competition Pass@1000":"8.80%","Competition Pass@5":"0.00%","Competition Pass@any":"8.80%","Interview Pass@1":"0.70%","Interview Pass@1000":"9.27%","Interview Pass@5":"1.03%","Interview Pass@any":"9.27%","Introductory Pass@1":"1.30%","Introductory Pass@1000":"25.00%","Introductory Pass@5":"3.60%","Introductory Pass@any":"25.00%"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-codecontests","task":"Code Generation","dataset":"CodeContests","model":"MapCoder (GPT-4)","rank_in_archive_order":3,"of":8,"metrics":{"Test Set pass@1":"28.5","Test Set pass@5":"35.2","Val Set pass@1":"28.5"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-humaneval","task":"Code Generation","dataset":"HumanEval","model":"Mistral 7B","rank_in_archive_order":6,"of":8,"metrics":{"Pass@1":"93.9"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-mbpp","task":"Code Generation","dataset":"MBPP","model":"o1-mini + MapCoder (Hamming.ai)","rank_in_archive_order":3,"of":99,"metrics":{"Accuracy":"93.2"},"uses_additional_data":true},{"leaderboard":"/sota/code-generation-on-mbpp","task":"Code Generation","dataset":"MBPP","model":"MapCoder (GPT-4o)","rank_in_archive_order":9,"of":99,"metrics":{"Accuracy":"89.7"},"uses_additional_data":false},{"leaderboard":"/sota/code-generation-on-mbpp","task":"Code Generation","dataset":"MBPP","model":"MapCoder (GPT-4)","rank_in_archive_order":17,"of":99,"metrics":{"Accuracy":"83.1"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.11403","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.11403"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Luoji-zju/Agents4PLC_release","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/md-ashraful-pramanik/mapcoder","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":10},"by_repo_kind":{"listed":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"de3a0618a914e97e","entry":"create_log_directory","repo":"Luoji-zju/Agents4PLC_release","repo_kind":"listed","path":"LLM4PLC_reproduce/run_efficiency.py","file_url":"https://github.com/Luoji-zju/Agents4PLC_release/blob/HEAD/LLM4PLC_reproduce/run_efficiency.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"de3a0618a914e97e"}},{"code_sha256_prefix":"464c0c08fefdf90c","entry":"generate_nl_description","repo":"Luoji-zju/Agents4PLC_release","repo_kind":"listed","path":"src/plcverif.py","file_url":"https://github.com/Luoji-zju/Agents4PLC_release/blob/HEAD/src/plcverif.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"464c0c08fefdf90c"}},{"code_sha256_prefix":"fe740cfb7327d8b2","entry":"input_with_timeout","repo":"Luoji-zju/Agents4PLC_release","repo_kind":"listed","path":"LLM4PLC_reproduce/run_efficiency.py","file_url":"https://github.com/Luoji-zju/Agents4PLC_release/blob/HEAD/LLM4PLC_reproduce/run_efficiency.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe740cfb7327d8b2"}},{"code_sha256_prefix":"b4ae4f09792713f2","entry":"input_with_timeout","repo":"Luoji-zju/Agents4PLC_release","repo_kind":"listed","path":"LLM4PLC_reproduce/run_efficiency_v2.py","file_url":"https://github.com/Luoji-zju/Agents4PLC_release/blob/HEAD/LLM4PLC_reproduce/run_efficiency_v2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4ae4f09792713f2"}},{"code_sha256_prefix":"b20bd1bc19f9f7df","entry":"list_to_json_string","repo":"Luoji-zju/Agents4PLC_release","repo_kind":"listed","path":"src/tools.py","file_url":"https://github.com/Luoji-zju/Agents4PLC_release/blob/HEAD/src/tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b20bd1bc19f9f7df"}},{"code_sha256_prefix":"848bf700245e1978","entry":"nuXmv_model_checker","repo":"Luoji-zju/Agents4PLC_release","repo_kind":"listed","path":"LLM4PLC_reproduce/run_efficiency.py","file_url":"https://github.com/Luoji-zju/Agents4PLC_release/blob/HEAD/LLM4PLC_reproduce/run_efficiency.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"848bf700245e1978"}},{"code_sha256_prefix":"d76fc1eb310fdfa3","entry":"nuXmv_model_checker","repo":"Luoji-zju/Agents4PLC_release","repo_kind":"listed","path":"LLM4PLC_reproduce/run_efficiency_v2.py","file_url":"https://github.com/Luoji-zju/Agents4PLC_release/blob/HEAD/LLM4PLC_reproduce/run_efficiency_v2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d76fc1eb310fdfa3"}},{"code_sha256_prefix":"a632c1e691ea6a16","entry":"plcverif_call","repo":"Luoji-zju/Agents4PLC_release","repo_kind":"listed","path":"src/plcverif.py","file_url":"https://github.com/Luoji-zju/Agents4PLC_release/blob/HEAD/src/plcverif.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a632c1e691ea6a16"}},{"code_sha256_prefix":"f8a97b7fd226ab76","entry":"rusty_compiler","repo":"Luoji-zju/Agents4PLC_release","repo_kind":"listed","path":"src/compiler.py","file_url":"https://github.com/Luoji-zju/Agents4PLC_release/blob/HEAD/src/compiler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8a97b7fd226ab76"}},{"code_sha256_prefix":"e00def812098236a","entry":"safe_repr","repo":"Luoji-zju/Agents4PLC_release","repo_kind":"listed","path":"src/tools.py","file_url":"https://github.com/Luoji-zju/Agents4PLC_release/blob/HEAD/src/tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e00def812098236a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}