{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tdd-without-tears-towards-test-case","title":"PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation","arxiv_id":"2401.07576","date":"2024-01-15","proceeding":null,"authors":["Wannita Takerngsaksiri","Rujikorn Charakorn","Chakkrit Tantithamthavorn","Yuan-Fang Li"],"abstract":"Test-driven development (TDD) is a widely-employed software development practice that mandates writing test cases based on requirements before writing the actual code. While writing test cases is the centerpiece of TDD, it is time-consuming, expensive, and often shunned by developers. To address these issues associated with TDD, automated test case generation approaches have recently been investigated. Such approaches take source code as input, but not the requirements. Therefore, existing work does not fully support true TDD, as actual code is required to generate test cases. In addition, current deep learning-based test case generation approaches are trained with one learning objective, i.e., to generate test cases that are exactly matched with the ground-truth test cases. However, such approaches may limit the model's ability to generate different yet correct test cases. In this paper, we introduce PyTester, a Text-to-Testcase generation approach that can automatically generate syntactically correct, executable, complete, and effective test cases while being aligned with a given natural language requirement. We evaluate PyTester on the public APPS benchmark dataset, and the results show that our Deep RL approach enables PyTester, a small language model, to outperform much larger language models like GPT3.5, StarCoder, and InCoder. Our findings suggest that future research could consider improving small over large LMs for better resource efficiency by integrating the SE domain knowledge into the design of reinforcement learning architecture.","url_abs":"https://arxiv.org/abs/2401.07576v2","url_pdf":"https://arxiv.org/pdf/2401.07576v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"tdd-without-tears-towards-test-case","repo_url":"https://github.com/tddpytester/pytester","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2401.07576","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.07576"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tddpytester/pytester","reach":{"status":"ok"}}],"summary":{"ran":8,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"d4025094cd1b7d1b","entry":"check_test_case_syntax","repo":"tddpytester/pytester","repo_kind":"official","path":"handlers/testing_util.py","file_url":"https://github.com/tddpytester/pytester/blob/HEAD/handlers/testing_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d4025094cd1b7d1b"}},{"code_sha256_prefix":"70db7c9cbe9e680d","entry":"load_pickle","repo":"tddpytester/pytester","repo_kind":"official","path":"handlers/utils.py","file_url":"https://github.com/tddpytester/pytester/blob/HEAD/handlers/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70db7c9cbe9e680d"}},{"code_sha256_prefix":"c479a69c9318680d","entry":"post","repo":"tddpytester/pytester","repo_kind":"official","path":"baselines/Copilot/query_apps.py","file_url":"https://github.com/tddpytester/pytester/blob/HEAD/baselines/Copilot/query_apps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c479a69c9318680d"}},{"code_sha256_prefix":"bfa04064fa4b10d8","entry":"read_json_file","repo":"tddpytester/pytester","repo_kind":"official","path":"handlers/utils.py","file_url":"https://github.com/tddpytester/pytester/blob/HEAD/handlers/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfa04064fa4b10d8"}},{"code_sha256_prefix":"fa42b92f92918cb1","entry":"split_test_cases","repo":"tddpytester/pytester","repo_kind":"official","path":"handlers/testing_util.py","file_url":"https://github.com/tddpytester/pytester/blob/HEAD/handlers/testing_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa42b92f92918cb1"}},{"code_sha256_prefix":"48927d42c199464b","entry":"split_test_cases","repo":"tddpytester/pytester","repo_kind":"official","path":"handlers/testing_util_v2.py","file_url":"https://github.com/tddpytester/pytester/blob/HEAD/handlers/testing_util_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48927d42c199464b"}},{"code_sha256_prefix":"cd5ea938ecaf9b22","entry":"test_function","repo":"tddpytester/pytester","repo_kind":"official","path":"handlers/testing_util.py","file_url":"https://github.com/tddpytester/pytester/blob/HEAD/handlers/testing_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd5ea938ecaf9b22"}},{"code_sha256_prefix":"23e99f1aaaf91949","entry":"transform_to_code_and_test","repo":"tddpytester/pytester","repo_kind":"official","path":"handlers/testing_util_v2.py","file_url":"https://github.com/tddpytester/pytester/blob/HEAD/handlers/testing_util_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23e99f1aaaf91949"}},{"code_sha256_prefix":"76a2b0b9d12a137d","entry":"transform_to_input","repo":"tddpytester/pytester","repo_kind":"official","path":"handlers/testing_util_v2.py","file_url":"https://github.com/tddpytester/pytester/blob/HEAD/handlers/testing_util_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76a2b0b9d12a137d"}},{"code_sha256_prefix":"e47c17a9039c79db","entry":"DecodeIds","repo":"tddpytester/pytester","repo_kind":"official","path":"handlers/code_processing.py","file_url":"https://github.com/tddpytester/pytester/blob/HEAD/handlers/code_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e47c17a9039c79db"}},{"code_sha256_prefix":"0cba2737fdabc294","entry":"copilot","repo":"tddpytester/pytester","repo_kind":"official","path":"baselines/Copilot/api.py","file_url":"https://github.com/tddpytester/pytester/blob/HEAD/baselines/Copilot/api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0cba2737fdabc294"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}