{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/effective-test-generation-using-pre-trained","title":"Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing","arxiv_id":"2308.16557","date":"2023-08-31","proceeding":null,"authors":["Arghavan Moradi Dakhel","Amin Nikanjam","Vahid Majdinasab","Foutse Khomh","Michel C. Desmarais"],"abstract":"One of the critical phases in software development is software testing. Testing helps with identifying potential bugs and reducing maintenance costs. The goal of automated test generation tools is to ease the development of tests by suggesting efficient bug-revealing tests. Recently, researchers have leveraged Large Language Models (LLMs) of code to generate unit tests. While the code coverage of generated tests was usually assessed, the literature has acknowledged that the coverage is weakly correlated with the efficiency of tests in bug detection. To improve over this limitation, in this paper, we introduce MuTAP for improving the effectiveness of test cases generated by LLMs in terms of revealing bugs by leveraging mutation testing. Our goal is achieved by augmenting prompts with surviving mutants, as those mutants highlight the limitations of test cases in detecting bugs. MuTAP is capable of generating effective test cases in the absence of natural language descriptions of the Program Under Test (PUTs). We employ different LLMs within MuTAP and evaluate their performance on different benchmarks. Our results show that our proposed method is able to detect up to 28% more faulty human-written code snippets. Among these, 17% remained undetected by both the current state-of-the-art fully automated test generation tool (i.e., Pynguin) and zero-shot/few-shot learning approaches on LLMs. Furthermore, MuTAP achieves a Mutation Score (MS) of 93.57% on synthetic buggy code, outperforming all other approaches in our evaluation. Our findings suggest that although LLMs can serve as a useful tool to generate test cases, they require specific post-processing steps to enhance the effectiveness of the generated test cases which may suffer from syntactic or functional errors and may be ineffective in detecting certain types of bugs and testing corner cases PUTs.","url_abs":"https://arxiv.org/abs/2308.16557v1","url_pdf":"https://arxiv.org/pdf/2308.16557v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"effective-test-generation-using-pre-trained","repo_url":"https://github.com/expertisemodel/mutap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.16557","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.16557"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/expertisemodel/mutap","reach":{"status":"ok"}}],"summary":{"ran":14,"unverified":2},"by_repo_kind":{"official":{"samples":16,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"3d640759c588109a","entry":"Mut_all_in_one","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/Merge_all_mut.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/Merge_all_mut.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d640759c588109a"}},{"code_sha256_prefix":"e755516d4ffe111d","entry":"Write_into_pickle","repo":"expertisemodel/mutap","repo_kind":"official","path":"Read_json_data.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/Read_json_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e755516d4ffe111d"}},{"code_sha256_prefix":"a5bc80a559b5038c","entry":"check_test_oracle_sematic","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/Sematic_err_correction.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/Sematic_err_correction.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5bc80a559b5038c"}},{"code_sha256_prefix":"ce15f08dfed3da2d","entry":"directory_crawler","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/greedy_test_generator.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/greedy_test_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce15f08dfed3da2d"}},{"code_sha256_prefix":"54591bb7dd8a8e60","entry":"fewshotMutantPromptGenerator","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/augmented_prompt.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/augmented_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54591bb7dd8a8e60"}},{"code_sha256_prefix":"f43c94b84ab25a2c","entry":"fewshotPromptGenerator","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/generate_test_oracle.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/generate_test_oracle.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f43c94b84ab25a2c"}},{"code_sha256_prefix":"2a493a7d68af773f","entry":"initalPromptGenerator","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/generate_test_oracle.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/generate_test_oracle.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a493a7d68af773f"}},{"code_sha256_prefix":"bfc6d0ec102ca108","entry":"load_model","repo":"expertisemodel/mutap","repo_kind":"official","path":"llama_util/model_utils.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/llama_util/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfc6d0ec102ca108"}},{"code_sha256_prefix":"0bd08159faf7b105","entry":"merge_asserts","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/Merge_all_mut.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/Merge_all_mut.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bd08159faf7b105"}},{"code_sha256_prefix":"10e879155a44ead2","entry":"mutatePromptGenerator","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/augmented_prompt.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/augmented_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10e879155a44ead2"}},{"code_sha256_prefix":"4e2d817f03e60189","entry":"paths_list","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/Mutation_Score.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/Mutation_Score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e2d817f03e60189"}},{"code_sha256_prefix":"77d3223b6589a4e6","entry":"read_csv","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/augmented_prompt.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/augmented_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77d3223b6589a4e6"}},{"code_sha256_prefix":"f7c84817041483b2","entry":"separate_asserts","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/greedy_test_generator.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/greedy_test_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7c84817041483b2"}},{"code_sha256_prefix":"ba4811601ac7b530","entry":"synxPromptGenerator","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/generate_test_oracle.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/generate_test_oracle.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba4811601ac7b530"}},{"code_sha256_prefix":"c129303ba93e3e99","entry":"apply_semantix_fix","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/Sematic_err_correction.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/Sematic_err_correction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c129303ba93e3e99"}},{"code_sha256_prefix":"7574ac527059c5f2","entry":"run_asserts_on_scripts","repo":"expertisemodel/mutap","repo_kind":"official","path":"MuTAP/greedy_test_generator.py","file_url":"https://github.com/expertisemodel/mutap/blob/HEAD/MuTAP/greedy_test_generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7574ac527059c5f2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}