{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-large-language-models-as-ai","title":"MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation","arxiv_id":"2310.03302","date":"2023-10-05","proceeding":null,"authors":["Qian Huang","Jian Vora","Percy Liang","Jure Leskovec"],"abstract":"A central aspect of machine learning research is experimentation, the process of designing and running experiments, analyzing the results, and iterating towards some positive outcome (e.g., improving accuracy). Could agents driven by powerful language models perform machine learning experimentation effectively? To answer this question, we introduce MLAgentBench, a suite of 13 tasks ranging from improving model performance on CIFAR-10 to recent research problems like BabyLM. For each task, an agent can perform actions like reading/writing files, executing code, and inspecting outputs. We then construct an agent that can perform ML experimentation based on ReAct framework. We benchmark agents based on Claude v1.0, Claude v2.1, Claude v3 Opus, GPT-4, GPT-4-turbo, Gemini-Pro, and Mixtral and find that a Claude v3 Opus agent is the best in terms of success rate. It can build compelling ML models over many tasks in MLAgentBench with 37.5% average success rate. Our agents also display highly interpretable plans and actions. However, the success rates vary considerably; they span from 100% on well-established older datasets to as low as 0% on recent Kaggle challenges created potentially after the underlying LM was trained. Finally, we identify several key challenges for LM-based agents such as long-term planning and reducing hallucination. Our code is released at https://github.com/snap-stanford/MLAgentBench.","url_abs":"https://arxiv.org/abs/2310.03302v2","url_pdf":"https://arxiv.org/pdf/2310.03302v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-large-language-models-as-ai","repo_url":"https://github.com/snap-stanford/mlagentbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"benchmarking-large-language-models-as-ai","repo_url":"https://github.com/zeroxleo/ml-agent","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"hallucination","task_name":"Hallucination"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.03302","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.03302"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zeroxleo/ml-agent","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/snap-stanford/mlagentbench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8fbba13275be664e","entry":"check_file_read_only","repo":"snap-stanford/mlagentbench","repo_kind":"official","path":"MLAgentBench/low_level_actions.py","file_url":"https://github.com/snap-stanford/mlagentbench/blob/HEAD/MLAgentBench/low_level_actions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fbba13275be664e"}},{"code_sha256_prefix":"53250bc9bc9a4820","entry":"mkl_error","repo":"snap-stanford/mlagentbench","repo_kind":"official","path":"MLAgentBench/plot.py","file_url":"https://github.com/snap-stanford/mlagentbench/blob/HEAD/MLAgentBench/plot.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53250bc9bc9a4820"}},{"code_sha256_prefix":"357dc7ee438a7576","entry":"oom_error","repo":"snap-stanford/mlagentbench","repo_kind":"official","path":"MLAgentBench/plot.py","file_url":"https://github.com/snap-stanford/mlagentbench/blob/HEAD/MLAgentBench/plot.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"357dc7ee438a7576"}},{"code_sha256_prefix":"4f82a282dda31635","entry":"reflection","repo":"snap-stanford/mlagentbench","repo_kind":"official","path":"MLAgentBench/high_level_actions.py","file_url":"https://github.com/snap-stanford/mlagentbench/blob/HEAD/MLAgentBench/high_level_actions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f82a282dda31635"}},{"code_sha256_prefix":"e78c16077ff8eccc","entry":"understand_file","repo":"snap-stanford/mlagentbench","repo_kind":"official","path":"MLAgentBench/high_level_actions.py","file_url":"https://github.com/snap-stanford/mlagentbench/blob/HEAD/MLAgentBench/high_level_actions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e78c16077ff8eccc"}},{"code_sha256_prefix":"28af11cb458eb742","entry":"create_benchmark_folder_name","repo":"snap-stanford/mlagentbench","repo_kind":"official","path":"MLAgentBench/environment.py","file_url":"https://github.com/snap-stanford/mlagentbench/blob/HEAD/MLAgentBench/environment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"28af11cb458eb742"}},{"code_sha256_prefix":"a1cb2a52e65ff361","entry":"edit_script","repo":"snap-stanford/mlagentbench","repo_kind":"official","path":"MLAgentBench/high_level_actions.py","file_url":"https://github.com/snap-stanford/mlagentbench/blob/HEAD/MLAgentBench/high_level_actions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1cb2a52e65ff361"}},{"code_sha256_prefix":"9c55f0596ab5430b","entry":"estimate_tokens","repo":"snap-stanford/mlagentbench","repo_kind":"official","path":"MLAgentBench/plot.py","file_url":"https://github.com/snap-stanford/mlagentbench/blob/HEAD/MLAgentBench/plot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c55f0596ab5430b"}},{"code_sha256_prefix":"2868580e4a832fe7","entry":"get_task_info","repo":"snap-stanford/mlagentbench","repo_kind":"official","path":"MLAgentBench/prepare_task.py","file_url":"https://github.com/snap-stanford/mlagentbench/blob/HEAD/MLAgentBench/prepare_task.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2868580e4a832fe7"}},{"code_sha256_prefix":"d6d0c6d7bde0bdcc","entry":"normalize_args_kwargs","repo":"snap-stanford/mlagentbench","repo_kind":"official","path":"MLAgentBench/low_level_actions.py","file_url":"https://github.com/snap-stanford/mlagentbench/blob/HEAD/MLAgentBench/low_level_actions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6d0c6d7bde0bdcc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}