{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-21652","title":"AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite","arxiv_id":"2510.21652","date":"2025-10-24","proceeding":"ICLR","authors":["Jonathan Bragg","Mike D'Arcy","Nishant Balepur","Dan Bareket","Bhavana Dalvi","Sergey Feldman","Dany Haddad","Jena D. Hwang","Peter Jansen","Varsha Kishore","Bodhisattwa Prasad Majumder","Aakanksha Naik","Sigal Rahamimov","Kyle Richardson","Amanpreet Singh","Harshit Surana","Aryeh Tiktinsky","Rosni Vasu","Guy Wiener","Chloe Anastasiades","Stefan Candra","Jason Dunkelberger","Dan Emery","Rob Evans","Malachi Hamada","Regan Huff","Rodney Kinney","Matt Latzke","Jaron Lochner","Ruben Lozano-Aguilera","Cecile Nguyen","Smita Rao","Amber Tanaka","Brooke Vlahos","Peter Clark","Doug Downey","Yoav Goldberg","Ashish Sabharwal","Daniel S. Weld"],"abstract":"AI agents hold the potential to revolutionize scientific productivity by automating literature reviews, replicating experiments, analyzing data, and even proposing new directions of inquiry; indeed, there are now many such agents, ranging from general-purpose \"deep research\" systems to specialized science-specific agents, such as AI Scientist and AIGS. Rigorous evaluation of these agents is critical for progress. Yet existing benchmarks fall short on several fronts: they often (1) lack reproducible agent tools necessary for a controlled comparison of core agentic capabilities; (2) do not account for confounding variables such as model cost and tool access; (3) do not provide standardized interfaces for quick agent prototyping and evaluation; (4) fail to provide holistic, product-informed measures of real-world use cases such as science research; and (5) lack comprehensive baseline agents necessary to identify true advances. In response, we define principles and tooling for more rigorously benchmarking agents. Using these, we present AstaBench, a suite that provides a holistic measure of agentic ability to perform scientific research, comprising 2400+ problems spanning the entire scientific discovery process and multiple scientific domains, and including many problems inspired by actual user requests to deployed Asta agents. Our suite comes with the first scientific research environment with production-grade search tools that enable controlled, reproducible evaluation, better accounting for confounders. Alongside, we provide a comprehensive suite of nine science-optimized classes of Asta agents and numerous baselines. Our extensive evaluation of 57 agents across 22 agent classes reveals several interesting findings, most importantly that despite meaningful progress on certain individual aspects, AI remains far from solving the challenge of science research assistance.","url_abs":"https://arxiv.org/abs/2510.21652","url_pdf":"https://arxiv.org/pdf/2510.21652","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.21652","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.21652"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/allenai/agent-baselines","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":5},"by_repo_kind":{"found_in_text":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2a87e111db4f16b1","entry":"clean_observation","repo":"allenai/agent-baselines","repo_kind":"found_in_text","path":"agent_baselines/solvers/code_agent/llm_agent.py","file_url":"https://github.com/allenai/agent-baselines/blob/HEAD/agent_baselines/solvers/code_agent/llm_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a87e111db4f16b1"}},{"code_sha256_prefix":"d2a7a3788d174821","entry":"delete_lines_containing","repo":"allenai/agent-baselines","repo_kind":"found_in_text","path":"agent_baselines/solvers/code_agent/llm_agent.py","file_url":"https://github.com/allenai/agent-baselines/blob/HEAD/agent_baselines/solvers/code_agent/llm_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d2a7a3788d174821"}},{"code_sha256_prefix":"543b1d7a11074f11","entry":"find_matches","repo":"allenai/agent-baselines","repo_kind":"found_in_text","path":"agent_baselines/solvers/code_agent/environment.py","file_url":"https://github.com/allenai/agent-baselines/blob/HEAD/agent_baselines/solvers/code_agent/environment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"543b1d7a11074f11"}},{"code_sha256_prefix":"cdad0f257db7572a","entry":"parse_response","repo":"allenai/agent-baselines","repo_kind":"found_in_text","path":"agent_baselines/solvers/code_agent/llm_agent.py","file_url":"https://github.com/allenai/agent-baselines/blob/HEAD/agent_baselines/solvers/code_agent/llm_agent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdad0f257db7572a"}},{"code_sha256_prefix":"6c50b73695bcfe90","entry":"query_youcom","repo":"allenai/agent-baselines","repo_kind":"found_in_text","path":"agent_baselines/solvers/youcom.py","file_url":"https://github.com/allenai/agent-baselines/blob/HEAD/agent_baselines/solvers/youcom.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6c50b73695bcfe90"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.AI","source":"arxiv_api"},"syntology_extracted_results":null}