{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/execution-based-evaluation-for-data-science","title":"Execution-based Evaluation for Data Science Code Generation Models","arxiv_id":"2211.09374","date":"2022-11-17","proceeding":null,"authors":["JunJie Huang","Chenglong Wang","Jipeng Zhang","Cong Yan","Haotian Cui","Jeevana Priya Inala","Colin Clement","Nan Duan","Jianfeng Gao"],"abstract":"Code generation models can benefit data scientists' productivity by automatically generating code from context and text descriptions. An important measure of the modeling progress is whether a model can generate code that can correctly execute to solve the task. However, due to the lack of an evaluation dataset that directly supports execution-based model evaluation, existing work relies on code surface form similarity metrics (e.g., BLEU, CodeBLEU) for model selection, which can be inaccurate. To remedy this, we introduce ExeDS, an evaluation dataset for execution evaluation for data science code generation tasks. ExeDS contains a set of 534 problems from Jupyter Notebooks, each consisting of code context, task description, reference program, and the desired execution output. With ExeDS, we evaluate the execution performance of five state-of-the-art code generation models that have achieved high surface-form evaluation scores. Our experiments show that models with high surface-form scores do not necessarily perform well on execution metrics, and execution-based metrics can better capture model code generation errors. Source code and data can be found at https://github.com/Jun-jie-Huang/ExeDS","url_abs":"https://arxiv.org/abs/2211.09374v1","url_pdf":"https://arxiv.org/pdf/2211.09374v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"execution-based-evaluation-for-data-science","repo_url":"https://github.com/jun-jie-huang/exeds","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"form","task_name":"Form"},{"task_slug":"model-selection","task_name":"Model Selection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.09374","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.09374"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jun-jie-huang/exeds","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":8},"by_repo_kind":{"official":{"samples":9,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2dd8e40379d7bbb9","entry":"compute_bleu","repo":"jun-jie-huang/exeds","repo_kind":"official","path":"codegpt/bleu.py","file_url":"https://github.com/jun-jie-huang/exeds/blob/HEAD/codegpt/bleu.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2dd8e40379d7bbb9"}},{"code_sha256_prefix":"c0f115991e111fb2","entry":"compare_code_string","repo":"jun-jie-huang/exeds","repo_kind":"official","path":"evaluation/evaluate_execution.py","file_url":"https://github.com/jun-jie-huang/exeds/blob/HEAD/evaluation/evaluate_execution.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0f115991e111fb2"}},{"code_sha256_prefix":"01492a12b494c3a7","entry":"csv_reader","repo":"jun-jie-huang/exeds","repo_kind":"official","path":"evaluation/evaluate_execution.py","file_url":"https://github.com/jun-jie-huang/exeds/blob/HEAD/evaluation/evaluate_execution.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01492a12b494c3a7"}},{"code_sha256_prefix":"3176f5d86b899016","entry":"get_c_list","repo":"jun-jie-huang/exeds","repo_kind":"official","path":"preprocess/preprocess.py","file_url":"https://github.com/jun-jie-huang/exeds/blob/HEAD/preprocess/preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3176f5d86b899016"}},{"code_sha256_prefix":"15c1d79ba288aedc","entry":"post_process_gptneo_generaion","repo":"jun-jie-huang/exeds","repo_kind":"official","path":"gptneo/post_process.py","file_url":"https://github.com/jun-jie-huang/exeds/blob/HEAD/gptneo/post_process.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15c1d79ba288aedc"}},{"code_sha256_prefix":"81b829e58d63ebb4","entry":"read_json","repo":"jun-jie-huang/exeds","repo_kind":"official","path":"codegpt/dataset.py","file_url":"https://github.com/jun-jie-huang/exeds/blob/HEAD/codegpt/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81b829e58d63ebb4"}},{"code_sha256_prefix":"327db3228bc36b13","entry":"read_json","repo":"jun-jie-huang/exeds","repo_kind":"official","path":"evaluation/evaluate_execution.py","file_url":"https://github.com/jun-jie-huang/exeds/blob/HEAD/evaluation/evaluate_execution.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"327db3228bc36b13"}},{"code_sha256_prefix":"d8eb246a783ad9f4","entry":"read_jsonl","repo":"jun-jie-huang/exeds","repo_kind":"official","path":"preprocess/preprocess.py","file_url":"https://github.com/jun-jie-huang/exeds/blob/HEAD/preprocess/preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8eb246a783ad9f4"}},{"code_sha256_prefix":"1afe99f2959d1014","entry":"read_txt_last","repo":"jun-jie-huang/exeds","repo_kind":"official","path":"codegpt/post_process.py","file_url":"https://github.com/jun-jie-huang/exeds/blob/HEAD/codegpt/post_process.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1afe99f2959d1014"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}