{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/code-agents-are-state-of-the-art-software","title":"SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents","arxiv_id":"2406.12952","date":"2024-06-18","proceeding":null,"authors":["Niels Mündler","Mark Niklas Müller","Jingxuan He","Martin Vechev"],"abstract":"Rigorous software testing is crucial for developing and maintaining high-quality code, making automated test generation a promising avenue for both improving software quality and boosting the effectiveness of code generation methods. However, while code generation with Large Language Models (LLMs) is an extraordinarily active research area, test generation remains relatively unexplored. We address this gap and investigate the capability of LLM-based Code Agents to formalize user issues into test cases. To this end, we propose a novel benchmark based on popular GitHub repositories, containing real-world issues, ground-truth bug-fixes, and golden tests. We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed specifically for test generation. Further, as test generation is a similar but more structured task than code generation, it allows for a more fine-grained analysis using issue reproduction rate and coverage changes, providing a dual metric for analyzing systems designed for code repair. Finally, we find that generated tests are an effective filter for proposed code fixes, doubling the precision of SWE-Agent. We release all data and code at https://github.com/logic-star-ai/SWT-Bench","url_abs":"https://arxiv.org/abs/2406.12952v3","url_pdf":"https://arxiv.org/pdf/2406.12952v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"code-agents-are-state-of-the-art-software","repo_url":"https://github.com/logic-star-ai/swt-bench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"code-repair","task_name":"Code Repair"},{"task_slug":"software-testing","task_name":"software testing"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.12952","atlas_url":"https://app.syntology.ai/?focus=2406.12952","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12952"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/logic-star-ai/swt-bench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c75625cf5e5b8c6","entry":"exec_run_with_timeout","repo":"logic-star-ai/swt-bench","repo_kind":"official","path":"src/docker_utils.py","file_url":"https://github.com/logic-star-ai/swt-bench/blob/HEAD/src/docker_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c75625cf5e5b8c6"}},{"code_sha256_prefix":"91fa9aa4140534fd","entry":"get_dockerfile_base","repo":"logic-star-ai/swt-bench","repo_kind":"official","path":"src/dockerfiles.py","file_url":"https://github.com/logic-star-ai/swt-bench/blob/HEAD/src/dockerfiles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91fa9aa4140534fd"}},{"code_sha256_prefix":"c417415f881e5f80","entry":"get_dockerfile_env","repo":"logic-star-ai/swt-bench","repo_kind":"official","path":"src/dockerfiles.py","file_url":"https://github.com/logic-star-ai/swt-bench/blob/HEAD/src/dockerfiles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c417415f881e5f80"}},{"code_sha256_prefix":"2366db697fa83da9","entry":"get_dockerfile_instance","repo":"logic-star-ai/swt-bench","repo_kind":"official","path":"src/dockerfiles.py","file_url":"https://github.com/logic-star-ai/swt-bench/blob/HEAD/src/dockerfiles.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2366db697fa83da9"}},{"code_sha256_prefix":"488a532d865fc92e","entry":"swe_to_swt_instance","repo":"logic-star-ai/swt-bench","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/logic-star-ai/swt-bench/blob/HEAD/src/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"488a532d865fc92e"}},{"code_sha256_prefix":"551579b16737ab25","entry":"main","repo":"logic-star-ai/swt-bench","repo_kind":"official","path":"dataset/swt_bench_cross_validation.py","file_url":"https://github.com/logic-star-ai/swt-bench/blob/HEAD/dataset/swt_bench_cross_validation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"551579b16737ab25"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}