{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ai-agents-that-matter","title":"AI Agents That Matter","arxiv_id":"2407.01502","date":"2024-07-01","proceeding":null,"authors":["Sayash Kapoor","Benedikt Stroebl","Zachary S. Siegel","Nitya Nadgir","Arvind Narayanan"],"abstract":"AI agents are an exciting new research direction, and agent development is driven by benchmarks. Our analysis of current agent benchmarks and evaluation practices reveals several shortcomings that hinder their usefulness in real-world applications. First, there is a narrow focus on accuracy without attention to other metrics. As a result, SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains. Our focus on cost in addition to accuracy motivates the new goal of jointly optimizing the two metrics. We design and implement one such optimization, showing its potential to greatly reduce cost while maintaining accuracy. Second, the benchmarking needs of model and downstream developers have been conflated, making it hard to identify which agent would be best suited for a particular application. Third, many agent benchmarks have inadequate holdout sets, and sometimes none at all. This has led to agents that are fragile because they take shortcuts and overfit to the benchmark in various ways. We prescribe a principled framework for avoiding overfitting. Finally, there is a lack of standardization in evaluation practices, leading to a pervasive lack of reproducibility. We hope that the steps we introduce for addressing these shortcomings will spur the development of agents that are useful in the real world and not just accurate on benchmarks.","url_abs":"https://arxiv.org/abs/2407.01502v1","url_pdf":"https://arxiv.org/pdf/2407.01502v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ai-agents-that-matter","repo_url":"https://github.com/benediktstroebl/agent-evals","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.01502","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.01502"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/benediktstroebl/agent-evals","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4917a312cc817135","entry":"gpt3","repo":"benediktstroebl/agent-evals","repo_kind":"official","path":"LanguageAgentTreeSearch/webshop/models.py","file_url":"https://github.com/benediktstroebl/agent-evals/blob/HEAD/LanguageAgentTreeSearch/webshop/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4917a312cc817135"}},{"code_sha256_prefix":"3f71176782469ee7","entry":"read_json_file","repo":"benediktstroebl/agent-evals","repo_kind":"official","path":"NovelQA/novelQA-gpt-4.py","file_url":"https://github.com/benediktstroebl/agent-evals/blob/HEAD/NovelQA/novelQA-gpt-4.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f71176782469ee7"}},{"code_sha256_prefix":"6bf8e275f958a588","entry":"read_txt_file","repo":"benediktstroebl/agent-evals","repo_kind":"official","path":"NovelQA/novelQA-gpt-4.py","file_url":"https://github.com/benediktstroebl/agent-evals/blob/HEAD/NovelQA/novelQA-gpt-4.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bf8e275f958a588"}},{"code_sha256_prefix":"045dd45316d89fe9","entry":"truncate_and_combine","repo":"benediktstroebl/agent-evals","repo_kind":"official","path":"NovelQA/novelQA-gpt-4.py","file_url":"https://github.com/benediktstroebl/agent-evals/blob/HEAD/NovelQA/novelQA-gpt-4.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"045dd45316d89fe9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}