{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/swe-bench-can-language-models-resolve-real","title":"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?","arxiv_id":"2310.06770","date":"2023-10-10","proceeding":null,"authors":["Carlos E. Jimenez","John Yang","Alexander Wettig","Shunyu Yao","Kexin Pei","Ofir Press","Karthik Narasimhan"],"abstract":"Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere $1.96$% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.","url_abs":"https://arxiv.org/abs/2310.06770v3","url_pdf":"https://arxiv.org/pdf/2310.06770v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"swe-bench-can-language-models-resolve-real","repo_url":"https://github.com/TRAIS-Lab/dca-bench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"swe-bench-can-language-models-resolve-real","repo_url":"https://github.com/deepsoftwareanalytics/omnigirl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"swe-bench-can-language-models-resolve-real","repo_url":"https://github.com/mariushobbhahn/swebench-verified-mini","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"swe-bench-can-language-models-resolve-real","repo_url":"https://github.com/multi-swe-bench/multi-swe-bench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"swe-bench-can-language-models-resolve-real","repo_url":"https://github.com/princeton-nlp/SWE-bench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"swe-bench-can-language-models-resolve-real","repo_url":"https://github.com/swe-bench/swe-bench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"swe-bench-can-language-models-resolve-real","repo_url":"https://github.com/swe-rebench/swe-bench-fork","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"swe-bench-can-language-models-resolve-real","repo_url":"https://github.com/thomasjoshi/agents-never-forget","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"bug-fixing","task_name":"Bug fixing"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[{"slug":"swe-bench","name":"SWE-bench-lite","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.06770","atlas_url":"https://app.syntology.ai/?focus=2310.06770","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.06770"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/swe-bench/swe-bench","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/TRAIS-Lab/dca-bench","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepsoftwareanalytics/omnigirl","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/princeton-nlp/SWE-bench","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/swe-rebench/swe-bench-fork","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mariushobbhahn/swebench-verified-mini","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/multi-swe-bench/multi-swe-bench","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thomasjoshi/agents-never-forget","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"listed":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"63dc2b373a4f046a","entry":"div_by_zero","repo":"TRAIS-Lab/dca-bench","repo_kind":"listed","path":"pipeline/utils.py","file_url":"https://github.com/TRAIS-Lab/dca-bench/blob/HEAD/pipeline/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"63dc2b373a4f046a"}},{"code_sha256_prefix":"c5ce103778b4f740","entry":"get_user_input","repo":"TRAIS-Lab/dca-bench","repo_kind":"listed","path":"pipeline/utils.py","file_url":"https://github.com/TRAIS-Lab/dca-bench/blob/HEAD/pipeline/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c5ce103778b4f740"}},{"code_sha256_prefix":"dbf6237329c48a48","entry":"load_human_labels","repo":"TRAIS-Lab/dca-bench","repo_kind":"listed","path":"pipeline/get_input.py","file_url":"https://github.com/TRAIS-Lab/dca-bench/blob/HEAD/pipeline/get_input.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dbf6237329c48a48"}},{"code_sha256_prefix":"2aa2855ab0183cf3","entry":"load_human_labels","repo":"TRAIS-Lab/dca-bench","repo_kind":"listed","path":"pipeline/utils.py","file_url":"https://github.com/TRAIS-Lab/dca-bench/blob/HEAD/pipeline/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2aa2855ab0183cf3"}},{"code_sha256_prefix":"46e4f6908904fa08","entry":"make_label_file","repo":"TRAIS-Lab/dca-bench","repo_kind":"listed","path":"pipeline/models/webpage_hard.py","file_url":"https://github.com/TRAIS-Lab/dca-bench/blob/HEAD/pipeline/models/webpage_hard.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46e4f6908904fa08"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}