{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-09447","title":"SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents","arxiv_id":"2602.09447","date":"2026-02-10","proceeding":null,"authors":["Zhirui Zhang","Hongbo Zhang","Haoxiang Fei","Zhiyuan Bao","Yubin Chen","Zhengyu Lei","Ziyue Liu","Yixuan Sun","Mingkun Xiao","Zihang Ye","Yu Zhang","Hongcheng Zhu","Yuxiang Wen","Heung-Yeung Shum"],"abstract":"Although large language models (LLMs) have demonstrated impressive coding capabilities, their ability to autonomously build production-scale software from explicit specifications remains an open question. We introduce SWE-AGI, an open-source benchmark for evaluating end-to-end, specification-driven construction of software systems written in MoonBit. SWE-AGI tasks require LLM-based agents to implement parsers, interpreters, binary decoders, and SAT solvers strictly from authoritative standards and RFCs under a fixed API scaffold. Each task involves implementing 1,000-10,000 lines of core logic, corresponding to weeks or months of engineering effort for an experienced human developer. By leveraging the nascent MoonBit ecosystem, SWE-AGI minimizes data leakage, forcing agents to rely on long-horizon architectural reasoning rather than code retrieval. Across frontier models, gpt-5.3-codex achieves the best overall performance (solving 19/22 tasks, 86.4%), outperforming claude-opus-4.6 (15/22, 68.2%), and kimi-2.5 exhibits the strongest performance among open-source models. Performance degrades sharply with increasing task difficulty, particularly on hard, specification-intensive systems. Behavioral analysis further reveals that as codebases scale, code reading, rather than writing, becomes the dominant bottleneck in AI-assisted development. Overall, while specification-driven autonomous software engineering is increasingly viable, substantial challenges remain before it can reliably support production-scale development.","url_abs":"https://arxiv.org/abs/2602.09447","url_pdf":"https://arxiv.org/pdf/2602.09447","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2602.09447","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.09447"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/moonbitlang/SWE-AGI","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/moonbitlang/SWE-AGI-Eval","reach":null}],"summary":{"ran":6,"ran_violates":3,"ran_draft_wrong":3,"unverified":8},"by_repo_kind":{"found_in_text":{"samples":20,"ran":12,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0d26355a0593d1cd","entry":"_extract_test_block_from_text","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d26355a0593d1cd"}},{"code_sha256_prefix":"3e52f0603142fd16","entry":"_find_test_file","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e52f0603142fd16"}},{"code_sha256_prefix":"5523b61639d585a9","entry":"_format_failure_message","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5523b61639d585a9"}},{"code_sha256_prefix":"5f4997aeaa42de77","entry":"_is_selection_error","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f4997aeaa42de77"}},{"code_sha256_prefix":"b5f7c09f9a07afea","entry":"_parse_failure_jsonl","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5f7c09f9a07afea"}},{"code_sha256_prefix":"2d84a21c452eeb6f","entry":"_parse_test_summary","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d84a21c452eeb6f"}},{"code_sha256_prefix":"d47e662df259f5cf","entry":"copy_project","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d47e662df259f5cf"}},{"code_sha256_prefix":"e49ed3ddb63cda58","entry":"run_cdcl_test_streaming","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e49ed3ddb63cda58"}},{"code_sha256_prefix":"db5f1c7ce134271e","entry":"run_moon_build","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"db5f1c7ce134271e"}},{"code_sha256_prefix":"bdd2704909cc54c3","entry":"send_sse_comment","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdd2704909cc54c3"}},{"code_sha256_prefix":"c39d6db42bcfe568","entry":"send_sse_event","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c39d6db42bcfe568"}},{"code_sha256_prefix":"719368dd9fb84c7b","entry":"terminate_process_gracefully","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"719368dd9fb84c7b"}},{"code_sha256_prefix":"48b9834c9dd08085","entry":"collect_labels","repo":"moonbitlang/SWE-AGI-Eval","repo_kind":"found_in_text","path":"claude-opus-4.5/cdcl/try.py","file_url":"https://github.com/moonbitlang/SWE-AGI-Eval/blob/HEAD/claude-opus-4.5/cdcl/try.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"48b9834c9dd08085"}},{"code_sha256_prefix":"bd08390f95c0fe75","entry":"colorize","repo":"moonbitlang/SWE-AGI-Eval","repo_kind":"found_in_text","path":"claude-opus-4.5/cdcl/try.py","file_url":"https://github.com/moonbitlang/SWE-AGI-Eval/blob/HEAD/claude-opus-4.5/cdcl/try.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd08390f95c0fe75"}},{"code_sha256_prefix":"7a0a58d0e71fa07f","entry":"generate_moonbit","repo":"moonbitlang/SWE-AGI-Eval","repo_kind":"found_in_text","path":"deepseek-v3.2/hpack/parse_huffman.py","file_url":"https://github.com/moonbitlang/SWE-AGI-Eval/blob/HEAD/deepseek-v3.2/hpack/parse_huffman.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7a0a58d0e71fa07f"}},{"code_sha256_prefix":"52c4ad71a7b7b591","entry":"handle_proj_submit_streaming","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52c4ad71a7b7b591"}},{"code_sha256_prefix":"6c8362c015ee20dd","entry":"log","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6c8362c015ee20dd"}},{"code_sha256_prefix":"786d0d4d4541702b","entry":"register_process","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"786d0d4d4541702b"}},{"code_sha256_prefix":"f0c203f3c5165c98","entry":"run_moon_test","repo":"moonbitlang/SWE-AGI","repo_kind":"found_in_text","path":"docker/server.py","file_url":"https://github.com/moonbitlang/SWE-AGI/blob/HEAD/docker/server.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0c203f3c5165c98"}},{"code_sha256_prefix":"6fed0709f2c2de4c","entry":"status_symbol","repo":"moonbitlang/SWE-AGI-Eval","repo_kind":"found_in_text","path":"claude-opus-4.5/cdcl/try.py","file_url":"https://github.com/moonbitlang/SWE-AGI-Eval/blob/HEAD/claude-opus-4.5/cdcl/try.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6fed0709f2c2de4c"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}