{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-large-scale-empirical-study-on-fine-tuning","title":"A Large-scale Empirical Study on Fine-tuning Large Language Models for Unit Testing","arxiv_id":"2412.16620","date":"2024-12-21","proceeding":null,"authors":["Ye Shang","Quanjun Zhang","Chunrong Fang","Siqi Gu","Jianyi Zhou","Zhenyu Chen"],"abstract":"Unit testing plays a pivotal role in software development, improving software quality and reliability. However, generating effective test cases manually is time-consuming, prompting interest in unit testing research. Recently, Large Language Models (LLMs) have shown potential in various unit testing tasks, including test generation, assertion generation, and test evolution, but existing studies are limited in scope and lack a systematic evaluation of the effectiveness of LLMs. To bridge this gap, we present a large-scale empirical study on fine-tuning LLMs for unit testing. Our study involves three unit testing tasks, five benchmarks, eight evaluation metrics, and 37 popular LLMs across various architectures and sizes, consuming over 3,000 NVIDIA A100 GPU hours. We focus on three key research questions: (1) the performance of LLMs compared to state-of-the-art methods, (2) the impact of different factors on LLM performance, and (3) the effectiveness of fine-tuning versus prompt engineering. Our findings reveal that LLMs outperform existing state-of-the-art approaches on all three unit testing tasks across nearly all metrics, highlighting the potential of fine-tuning LLMs in unit testing tasks. Furthermore, large-scale, decoder-only models achieve the best results across tasks, while encoder-decoder models perform better under the same parameter scale. Additionally, the comparison of the performance between fine-tuning and prompt engineering approaches reveals the considerable potential capability of the prompt engineering approach in unit testing tasks. We then discuss the concerned issues on the test generation task, including data leakage issues, bug detection capabilities, and metrics comparisons. Finally, we further pinpoint carious practical guidelines for LLM-based approaches to unit testing tasks in the near future.","url_abs":"https://arxiv.org/abs/2412.16620v1","url_pdf":"https://arxiv.org/pdf/2412.16620v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"a-large-scale-empirical-study-on-fine-tuning","repo_url":"https://github.com/iSEngLab/LLM4UT_Empirical","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2412.16620","atlas_url":"https://app.syntology.ai/?focus=2412.16620","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.16620"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/iSEngLab/LLM4UT_Empirical","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":3,"ran_violates":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"456070524f8a6af5","entry":"generate_one","repo":"iSEngLab/LLM4UT_Empirical","repo_kind":"official","path":"Inference_Script/open_source.py","file_url":"https://github.com/iSEngLab/LLM4UT_Empirical/blob/HEAD/Inference_Script/open_source.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"456070524f8a6af5"}},{"code_sha256_prefix":"38d4c9dc25380d39","entry":"get_prompt_target","repo":"iSEngLab/LLM4UT_Empirical","repo_kind":"official","path":"Finetune_Script/decoder_ag_main.py","file_url":"https://github.com/iSEngLab/LLM4UT_Empirical/blob/HEAD/Finetune_Script/decoder_ag_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"38d4c9dc25380d39"}},{"code_sha256_prefix":"02171abe8492fcfd","entry":"read_jsonl","repo":"iSEngLab/LLM4UT_Empirical","repo_kind":"official","path":"Inference_Script/open_source.py","file_url":"https://github.com/iSEngLab/LLM4UT_Empirical/blob/HEAD/Inference_Script/open_source.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"02171abe8492fcfd"}},{"code_sha256_prefix":"4bcd5408545e929d","entry":"smooth","repo":"iSEngLab/LLM4UT_Empirical","repo_kind":"official","path":"Finetune_Script/decoder_ag_main.py","file_url":"https://github.com/iSEngLab/LLM4UT_Empirical/blob/HEAD/Finetune_Script/decoder_ag_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4bcd5408545e929d"}},{"code_sha256_prefix":"cadd56f46fae79b8","entry":"tokenize","repo":"iSEngLab/LLM4UT_Empirical","repo_kind":"official","path":"Finetune_Script/decoder_ag_main.py","file_url":"https://github.com/iSEngLab/LLM4UT_Empirical/blob/HEAD/Finetune_Script/decoder_ag_main.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cadd56f46fae79b8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}