{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/websuite-systematically-evaluating-why-web","title":"WebSuite: Systematically Evaluating Why Web Agents Fail","arxiv_id":"2406.01623","date":"2024-06-01","proceeding":null,"authors":["Eric Li","Jim Waldo"],"abstract":"We describe WebSuite, the first diagnostic benchmark for generalist web agents, designed to systematically evaluate why agents fail. Advances in AI have led to the rise of numerous web agents that autonomously operate a browser to complete tasks. However, most existing benchmarks focus on strictly measuring whether an agent can or cannot complete a task, without giving insight on why. In this paper, we 1) develop a taxonomy of web actions to facilitate identifying common failure patterns, and 2) create an extensible benchmark suite to assess agents' performance on our taxonomized actions. This benchmark suite consists of both individual tasks, such as clicking a button, and end-to-end tasks, such as adding an item to a cart, and is designed such that any failure of a task can be attributed directly to a failure of a specific web action. We evaluate two popular generalist web agents, one text-based and one multimodal, and identify unique weaknesses for each agent. Because WebSuite can disaggregate task failures into specific action failures, this enables granular identification of which UX flows an individual agent has trouble with and immediately highlights promising avenues for improvement. These findings highlight the need for more focused benchmarking on where web agents go wrong to effectively improve agents beyond their weaker performance today.","url_abs":"https://arxiv.org/abs/2406.01623v1","url_pdf":"https://arxiv.org/pdf/2406.01623v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"websuite-systematically-evaluating-why-web","repo_url":"https://github.com/erichli1/websuite","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"diagnostic","task_name":"Diagnostic"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.01623","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.01623"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/erichli1/websuite","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6487e6a6f8d2e989","entry":"flatten","repo":"erichli1/websuite","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/erichli1/websuite/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6487e6a6f8d2e989"}},{"code_sha256_prefix":"d93c1027ece2cf49","entry":"get_evals_dict","repo":"erichli1/websuite","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/erichli1/websuite/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d93c1027ece2cf49"}},{"code_sha256_prefix":"f79218d0f55a18cc","entry":"get_url","repo":"erichli1/websuite","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/erichli1/websuite/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f79218d0f55a18cc"}},{"code_sha256_prefix":"6bdf7d071d0b5d3b","entry":"matching_urls","repo":"erichli1/websuite","repo_kind":"official","path":"evaluation/e2e.py","file_url":"https://github.com/erichli1/websuite/blob/HEAD/evaluation/e2e.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bdf7d071d0b5d3b"}},{"code_sha256_prefix":"8149016ddf10fd89","entry":"evaluate_e2e_outputs","repo":"erichli1/websuite","repo_kind":"official","path":"evaluation/tests.py","file_url":"https://github.com/erichli1/websuite/blob/HEAD/evaluation/tests.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8149016ddf10fd89"}},{"code_sha256_prefix":"3d9d7c4aee6baa5c","entry":"evaluate_e2e_task_outputs","repo":"erichli1/websuite","repo_kind":"official","path":"evaluation/tests.py","file_url":"https://github.com/erichli1/websuite/blob/HEAD/evaluation/tests.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d9d7c4aee6baa5c"}},{"code_sha256_prefix":"4e1a46626034a53c","entry":"evaluate_ind_outputs","repo":"erichli1/websuite","repo_kind":"official","path":"evaluation/tests.py","file_url":"https://github.com/erichli1/websuite/blob/HEAD/evaluation/tests.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e1a46626034a53c"}},{"code_sha256_prefix":"57436e0fa4a87ad6","entry":"get_specific_test_and_metadata","repo":"erichli1/websuite","repo_kind":"official","path":"evaluation/ind.py","file_url":"https://github.com/erichli1/websuite/blob/HEAD/evaluation/ind.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57436e0fa4a87ad6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}