{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ds-agent-automated-data-science-by-empowering","title":"DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning","arxiv_id":"2402.17453","date":"2024-02-27","proceeding":null,"authors":["Siyuan Guo","Cheng Deng","Ying Wen","Hechang Chen","Yi Chang","Jun Wang"],"abstract":"In this work, we investigate the potential of large language models (LLMs) based agents to automate data science tasks, with the goal of comprehending task requirements, then building and training the best-fit machine learning models. Despite their widespread success, existing LLM agents are hindered by generating unreasonable experiment plans within this scenario. To this end, we present DS-Agent, a novel automatic framework that harnesses LLM agent and case-based reasoning (CBR). In the development stage, DS-Agent follows the CBR framework to structure an automatic iteration pipeline, which can flexibly capitalize on the expert knowledge from Kaggle, and facilitate consistent performance improvement through the feedback mechanism. Moreover, DS-Agent implements a low-resource deployment stage with a simplified CBR paradigm to adapt past successful solutions from the development stage for direct code generation, significantly reducing the demand on foundational capabilities of LLMs. Empirically, DS-Agent with GPT-4 achieves 100\\% success rate in the development stage, while attaining 36\\% improvement on average one pass rate across alternative LLMs in the deployment stage. In both stages, DS-Agent achieves the best rank in performance, costing \\$1.60 and \\$0.13 per run with GPT-4, respectively. Our data and code are open-sourced at https://github.com/guosyjlu/DS-Agent.","url_abs":"https://arxiv.org/abs/2402.17453v5","url_pdf":"https://arxiv.org/pdf/2402.17453v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ds-agent-automated-data-science-by-empowering","repo_url":"https://github.com/guosyjlu/ds-agent","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.17453","atlas_url":"https://app.syntology.ai/?focus=2402.17453","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.17453"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/guosyjlu/DS-Agent","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/guosyjlu/ds-agent","reach":{"status":"ok"}}],"summary":{"ran":9,"unverified":4},"by_repo_kind":{"official":{"samples":13,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"e48536a39e307f1c","entry":"compute_metrics_for_classification","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/experience_replay/chatgpt-prompt.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/experience_replay/chatgpt-prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e48536a39e307f1c"}},{"code_sha256_prefix":"c7c62555e1f51544","entry":"compute_metrics_for_classification","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/experience_replay/enzyme-substrate.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/experience_replay/enzyme-substrate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7c62555e1f51544"}},{"code_sha256_prefix":"34caab810cace413","entry":"execute_script","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/execution.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/execution.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34caab810cace413"}},{"code_sha256_prefix":"6419d62b28f3cfc1","entry":"predict","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/experience_replay/airline-reviews.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/experience_replay/airline-reviews.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6419d62b28f3cfc1"}},{"code_sha256_prefix":"d168412c0d94b909","entry":"predict","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/experience_replay/chatgpt-prompt.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/experience_replay/chatgpt-prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d168412c0d94b909"}},{"code_sha256_prefix":"a43e675ac4b0fed0","entry":"predict","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/experience_replay/enzyme-substrate.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/experience_replay/enzyme-substrate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a43e675ac4b0fed0"}},{"code_sha256_prefix":"0755b9a6ac3a5909","entry":"train_model","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/experience_replay/airline-reviews.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/experience_replay/airline-reviews.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0755b9a6ac3a5909"}},{"code_sha256_prefix":"53971733e3acbbb1","entry":"train_model","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/experience_replay/chatgpt-prompt.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/experience_replay/chatgpt-prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"53971733e3acbbb1"}},{"code_sha256_prefix":"e759f3555c9dc70d","entry":"train_model","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/experience_replay/enzyme-substrate.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/experience_replay/enzyme-substrate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e759f3555c9dc70d"}},{"code_sha256_prefix":"2b03d09920c8d2a1","entry":"compute_metrics_for_regression","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/experience_replay/airline-reviews.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/experience_replay/airline-reviews.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b03d09920c8d2a1"}},{"code_sha256_prefix":"6d0f8f6dd1bc8b2f","entry":"get_case","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/prompt.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d0f8f6dd1bc8b2f"}},{"code_sha256_prefix":"15c1e503e527a5bf","entry":"get_prompt","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/prompt.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15c1e503e527a5bf"}},{"code_sha256_prefix":"a45eb6f9978ac8b0","entry":"get_task","repo":"guosyjlu/DS-Agent","repo_kind":"official","path":"deployment/prompt.py","file_url":"https://github.com/guosyjlu/DS-Agent/blob/HEAD/deployment/prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a45eb6f9978ac8b0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}