{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/large-language-models-for-software","title":"Large Language Models for Software Engineering: A Systematic Literature Review","arxiv_id":"2308.10620","date":"2023-08-21","proceeding":null,"authors":["Xinyi Hou","Yanjie Zhao","Yue Liu","Zhou Yang","Kailong Wang","Li Li","Xiapu Luo","David Lo","John Grundy","Haoyu Wang"],"abstract":"Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a systematic literature review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We select and analyze 395 research papers from January 2017 to January 2024 to answer four key research questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, preprocessing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and flagging promising areas for future study. Our artifacts are publicly available at https://github.com/xinyi-hou/LLM4SE_SLR.","url_abs":"https://arxiv.org/abs/2308.10620v6","url_pdf":"https://arxiv.org/pdf/2308.10620v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"large-language-models-for-software","repo_url":"https://github.com/xinyi-hou/llm4se_slr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"systematic-literature-review","task_name":"Systematic Literature Review"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.10620","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.10620"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xinyi-hou/llm4se_slr","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"eaa49ee3d8730674","entry":"find_keywords","repo":"xinyi-hou/llm4se_slr","repo_kind":"official","path":"tools/filter/FindVenues.py","file_url":"https://github.com/xinyi-hou/llm4se_slr/blob/HEAD/tools/filter/FindVenues.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eaa49ee3d8730674"}},{"code_sha256_prefix":"591e7f5d5c1d1d4c","entry":"set_venue","repo":"xinyi-hou/llm4se_slr","repo_kind":"official","path":"tools/filter/FindVenues.py","file_url":"https://github.com/xinyi-hou/llm4se_slr/blob/HEAD/tools/filter/FindVenues.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"591e7f5d5c1d1d4c"}},{"code_sha256_prefix":"6e9f832869041072","entry":"set_venue_lower","repo":"xinyi-hou/llm4se_slr","repo_kind":"official","path":"tools/filter/FindVenues.py","file_url":"https://github.com/xinyi-hou/llm4se_slr/blob/HEAD/tools/filter/FindVenues.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e9f832869041072"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}