{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-data-science-agents","title":"Benchmarking Data Science Agents","arxiv_id":"2402.17168","date":"2024-02-27","proceeding":null,"authors":["Yuge Zhang","Qiyang Jiang","Xingyu Han","Nan Chen","Yuqing Yang","Kan Ren"],"abstract":"In the era of data-driven decision-making, the complexity of data analysis necessitates advanced expertise and tools of data science, presenting significant challenges even for specialists. Large Language Models (LLMs) have emerged as promising aids as data science agents, assisting humans in data analysis and processing. Yet their practical efficacy remains constrained by the varied demands of real-world applications and complicated analytical process. In this paper, we introduce DSEval -- a novel evaluation paradigm, as well as a series of innovative benchmarks tailored for assessing the performance of these agents throughout the entire data science lifecycle. Incorporating a novel bootstrapped annotation method, we streamline dataset preparation, improve the evaluation coverage, and expand benchmarking comprehensiveness. Our findings uncover prevalent obstacles and provide critical insights to inform future advancements in the field.","url_abs":"https://arxiv.org/abs/2402.17168v1","url_pdf":"https://arxiv.org/pdf/2402.17168v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-data-science-agents","repo_url":"https://github.com/metacopilot/dseval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"decision-making","task_name":"Decision Making"}],"methods":[],"datasets_introduced":[{"slug":"dseval-exercise","name":"DSEval-Exercise","full_name":""},{"slug":"dseval","name":"DSEval-Kaggle","full_name":""},{"slug":"dseval-leetcode","name":"DSEval-LeetCode","full_name":""},{"slug":"dseval-so","name":"DSEval-SO","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.17168","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.17168"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/metacopilot/dseval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"80aaeed8b41d1da8","entry":"add_handler","repo":"metacopilot/dseval","repo_kind":"official","path":"dseval/logging.py","file_url":"https://github.com/metacopilot/dseval/blob/HEAD/dseval/logging.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80aaeed8b41d1da8"}},{"code_sha256_prefix":"eb53b42301057566","entry":"add_indent","repo":"metacopilot/dseval","repo_kind":"official","path":"dseval/utils.py","file_url":"https://github.com/metacopilot/dseval/blob/HEAD/dseval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb53b42301057566"}},{"code_sha256_prefix":"995319536acf66b7","entry":"available_results","repo":"metacopilot/dseval","repo_kind":"official","path":"dseval/browser.py","file_url":"https://github.com/metacopilot/dseval/blob/HEAD/dseval/browser.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"995319536acf66b7"}},{"code_sha256_prefix":"2b525d76f753e7a8","entry":"categorize_comparison_failure","repo":"metacopilot/dseval","repo_kind":"official","path":"dseval/validator.py","file_url":"https://github.com/metacopilot/dseval/blob/HEAD/dseval/validator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b525d76f753e7a8"}},{"code_sha256_prefix":"7af9f9ee08a52c95","entry":"exec_code","repo":"metacopilot/dseval","repo_kind":"official","path":"dseval/utils.py","file_url":"https://github.com/metacopilot/dseval/blob/HEAD/dseval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7af9f9ee08a52c95"}},{"code_sha256_prefix":"6a28b96a048d0343","entry":"get_temporary_file_name","repo":"metacopilot/dseval","repo_kind":"official","path":"dseval/utils.py","file_url":"https://github.com/metacopilot/dseval/blob/HEAD/dseval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a28b96a048d0343"}},{"code_sha256_prefix":"399a475723bfd916","entry":"validator_comments_to_verdict","repo":"metacopilot/dseval","repo_kind":"official","path":"dseval/validator.py","file_url":"https://github.com/metacopilot/dseval/blob/HEAD/dseval/validator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"399a475723bfd916"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}