{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/recode-robustness-evaluation-of-code","title":"ReCode: Robustness Evaluation of Code Generation Models","arxiv_id":"2212.10264","date":"2022-12-20","proceeding":null,"authors":["Shiqi Wang","Zheng Li","Haifeng Qian","Chenghao Yang","Zijian Wang","Mingyue Shang","Varun Kumar","Samson Tan","Baishakhi Ray","Parminder Bhatia","Ramesh Nallapati","Murali Krishna Ramanathan","Dan Roth","Bing Xiang"],"abstract":"Code generation models have achieved impressive performance. However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness properties, critical for user experience when deployed in real-life applications, are not well understood. Most existing works on robustness in text or code tasks have focused on classification, while robustness in generation tasks is an uncharted area and to date there is no comprehensive benchmark for robustness in code generation. In this paper, we propose ReCode, a comprehensive robustness evaluation benchmark for code generation models. We customize over 30 transformations specifically for code on docstrings, function and variable names, code syntax, and code format. They are carefully designed to be natural in real-life coding practice, preserve the original semantic meaning, and thus provide multifaceted assessments of a model's robustness performance. With human annotators, we verified that over 90% of the perturbed prompts do not alter the semantic meaning of the original prompt. In addition, we define robustness metrics for code generation models considering the worst-case behavior under each type of perturbation, taking advantage of the fact that executing the generated code can serve as objective evaluation. We demonstrate ReCode on SOTA models using HumanEval, MBPP, as well as function completion tasks derived from them. Interesting observations include: better robustness for CodeGen over InCoder and GPT-J; models are most sensitive to syntax perturbations; more challenging robustness evaluation on MBPP over HumanEval.","url_abs":"https://arxiv.org/abs/2212.10264v1","url_pdf":"https://arxiv.org/pdf/2212.10264v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"recode-robustness-evaluation-of-code","repo_url":"https://github.com/amazon-science/recode","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"recode-robustness-evaluation-of-code","repo_url":"https://github.com/frabbisw/robustextended","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"recode-robustness-evaluation-of-code","repo_url":"https://github.com/Mind23-2/MindCode-134","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"humaneval","task_name":"HumanEval"},{"task_slug":null,"task_name":"mbpp"}],"methods":[{"method_slug":"codegen","method_name":"CodeGen"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.10264","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.10264"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-science/recode","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Mind23-2/MindCode-134","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/frabbisw/robustextended","reach":null}],"summary":{"ran_honours":3,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c0c5be70122b5d4f","entry":"calculate_passatk","repo":"amazon-science/recode","repo_kind":"official","path":"run_robust.py","file_url":"https://github.com/amazon-science/recode/blob/HEAD/run_robust.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c0c5be70122b5d4f"}},{"code_sha256_prefix":"7c2585f7de759663","entry":"count_files_present_nonemtpy","repo":"amazon-science/recode","repo_kind":"official","path":"evaluate-public-models/evaluate_model.py","file_url":"https://github.com/amazon-science/recode/blob/HEAD/evaluate-public-models/evaluate_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7c2585f7de759663"}},{"code_sha256_prefix":"d459632bb421c25b","entry":"load_prompts","repo":"frabbisw/robustextended","repo_kind":"listed","path":"evalplus/evaluate_inputs.py","file_url":"https://github.com/frabbisw/robustextended/blob/HEAD/evalplus/evaluate_inputs.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d459632bb421c25b"}},{"code_sha256_prefix":"aabb21de20501ffe","entry":"read_json","repo":"amazon-science/recode","repo_kind":"official","path":"run_robust.py","file_url":"https://github.com/amazon-science/recode/blob/HEAD/run_robust.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aabb21de20501ffe"}},{"code_sha256_prefix":"ff62f37fda621b9e","entry":"read_passatk","repo":"amazon-science/recode","repo_kind":"official","path":"run_robust.py","file_url":"https://github.com/amazon-science/recode/blob/HEAD/run_robust.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff62f37fda621b9e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}