{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-unit-test-suggestions","title":"TOGA: A Neural Method for Test Oracle Generation","arxiv_id":"2109.09262","date":"2021-09-20","proceeding":null,"authors":["Elizabeth Dinella","Gabriel Ryan","Todd Mytkowicz","Shuvendu K. Lahiri"],"abstract":"Testing is widely recognized as an important stage of the software development lifecycle. Effective software testing can provide benefits such as bug finding, preventing regressions, and documentation. In terms of documentation, unit tests express a unit's intended functionality, as conceived by the developer. A test oracle, typically expressed as an condition, documents the intended behavior of a unit under a given test prefix. Synthesizing a functional test oracle is a challenging problem, as it must capture the intended functionality rather than the implemented functionality. In this paper, we propose TOGA (a neural method for Test Oracle GenerAtion), a unified transformer-based neural approach to infer both exceptional and assertion test oracles based on the context of the focal method. Our approach can handle units with ambiguous or missing documentation, and even units with a missing implementation. We evaluate our approach on both oracle inference accuracy and functional bug-finding. Our technique improves accuracy by 33\\% over existing oracle inference approaches, achieving 96\\% overall accuracy on a held out test dataset. Furthermore, we show that when integrated with a automated test generation tool (EvoSuite), our approach finds 57 real world bugs in large-scale Java programs, including 30 bugs that are not found by any other automated testing method in our evaluation.","url_abs":"https://arxiv.org/abs/2109.09262v2","url_pdf":"https://arxiv.org/pdf/2109.09262v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"neural-unit-test-suggestions","repo_url":"https://github.com/microsoft/toga","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2109.09262","atlas_url":"https://app.syntology.ai/?focus=2109.09262","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.09262"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/toga","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"unverified":21},"by_repo_kind":{"official":{"samples":22,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"eb725d5794b15f6b","entry":"accuracy","repo":"microsoft/toga","repo_kind":"official","path":"model/assertions/run_classifier.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/assertions/run_classifier.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb725d5794b15f6b"}},{"code_sha256_prefix":"15b03d85dc1e5392","entry":"acc_and_f1","repo":"microsoft/toga","repo_kind":"official","path":"model/assertions/utils.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/assertions/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15b03d85dc1e5392"}},{"code_sha256_prefix":"8efa0e429d75adda","entry":"bool_assert_to_equals","repo":"microsoft/toga","repo_kind":"official","path":"eval/gen_tests_from_metadata.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/eval/gen_tests_from_metadata.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8efa0e429d75adda"}},{"code_sha256_prefix":"302fca0ae159d4a6","entry":"checkout_project","repo":"microsoft/toga","repo_kind":"official","path":"eval/extract_tests.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/eval/extract_tests.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"302fca0ae159d4a6"}},{"code_sha256_prefix":"264bcac39d00a4e1","entry":"clean","repo":"microsoft/toga","repo_kind":"official","path":"model/exception_data.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/exception_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"264bcac39d00a4e1"}},{"code_sha256_prefix":"95c91a2b77c7fb96","entry":"convert_examples_to_features","repo":"microsoft/toga","repo_kind":"official","path":"model/assertions/utils.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/assertions/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"95c91a2b77c7fb96"}},{"code_sha256_prefix":"9a5769855cf4c041","entry":"diff","repo":"microsoft/toga","repo_kind":"official","path":"model/ranking.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/ranking.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a5769855cf4c041"}},{"code_sha256_prefix":"f052c69e11bdefba","entry":"get_active_bugs","repo":"microsoft/toga","repo_kind":"official","path":"eval/collect_test_results.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/eval/collect_test_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f052c69e11bdefba"}},{"code_sha256_prefix":"bd5b0111e170a7f5","entry":"get_block","repo":"microsoft/toga","repo_kind":"official","path":"model/exception_data.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/exception_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd5b0111e170a7f5"}},{"code_sha256_prefix":"a260552ed8c878b5","entry":"get_bug_catching_tests","repo":"microsoft/toga","repo_kind":"official","path":"eval/extract_bug_metadata.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/eval/extract_bug_metadata.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a260552ed8c878b5"}},{"code_sha256_prefix":"cb2f381214a3dc43","entry":"get_imports","repo":"microsoft/toga","repo_kind":"official","path":"eval/aggregate_test_cases.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/eval/aggregate_test_cases.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb2f381214a3dc43"}},{"code_sha256_prefix":"322a7d0ecccd0222","entry":"get_prefix","repo":"microsoft/toga","repo_kind":"official","path":"eval/gen_tests_from_metadata.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/eval/gen_tests_from_metadata.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"322a7d0ecccd0222"}},{"code_sha256_prefix":"b0740759b39327f1","entry":"get_project_layout","repo":"microsoft/toga","repo_kind":"official","path":"eval/collect_test_results.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/eval/collect_test_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0740759b39327f1"}},{"code_sha256_prefix":"a6787586150c671d","entry":"get_test_harness","repo":"microsoft/toga","repo_kind":"official","path":"eval/aggregate_test_cases.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/eval/aggregate_test_cases.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a6787586150c671d"}},{"code_sha256_prefix":"40f0e80317b58c01","entry":"get_worst_idx","repo":"microsoft/toga","repo_kind":"official","path":"model/ranking.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/ranking.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40f0e80317b58c01"}},{"code_sha256_prefix":"1af26ededf7d7d6e","entry":"insert_assertion","repo":"microsoft/toga","repo_kind":"official","path":"eval/gen_tests_from_metadata.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/eval/gen_tests_from_metadata.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1af26ededf7d7d6e"}},{"code_sha256_prefix":"3867c570f55a2b22","entry":"is_float","repo":"microsoft/toga","repo_kind":"official","path":"model/assertion_data.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/assertion_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3867c570f55a2b22"}},{"code_sha256_prefix":"1ed3568420522363","entry":"is_int","repo":"microsoft/toga","repo_kind":"official","path":"model/assertion_data.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/assertion_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ed3568420522363"}},{"code_sha256_prefix":"3f624daa44a57816","entry":"is_string","repo":"microsoft/toga","repo_kind":"official","path":"model/assertion_data.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/assertion_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f624daa44a57816"}},{"code_sha256_prefix":"b1e068a15b29a878","entry":"magnitude","repo":"microsoft/toga","repo_kind":"official","path":"model/ranking.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/ranking.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1e068a15b29a878"}},{"code_sha256_prefix":"c1793bf24d6a0bf4","entry":"normalize_assertThrows","repo":"microsoft/toga","repo_kind":"official","path":"model/exception_data.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/exception_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1793bf24d6a0bf4"}},{"code_sha256_prefix":"3c241ecfe3749a6d","entry":"simple_accuracy","repo":"microsoft/toga","repo_kind":"official","path":"model/assertions/utils.py","file_url":"https://github.com/microsoft/toga/blob/HEAD/model/assertions/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c241ecfe3749a6d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}