{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-browsergym-ecosystem-for-web-agent","title":"The BrowserGym Ecosystem for Web Agent Research","arxiv_id":"2412.05467","date":"2024-12-06","proceeding":null,"authors":["Thibault Le Sellier De Chezelles","Maxime Gasse","Alexandre Drouin","Massimo Caccia","Léo Boisvert","Megh Thakkar","Tom Marty","Rim Assouel","Sahar Omidi Shayegan","Lawrence Keunho Jang","Xing Han Lù","Ori Yoran","Dehan Kong","Frank F. Xu","Siva Reddy","Quentin Cappart","Graham Neubig","Ruslan Salakhutdinov","Nicolas Chapados","Alexandre Lacoste"],"abstract":"The BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents, particularly those leveraging automation and Large Language Models (LLMs). Many existing benchmarks suffer from fragmentation and inconsistent evaluation methodologies, making it challenging to achieve reliable comparisons and reproducible results. In an earlier work, Drouin et al. (2024) introduced BrowserGym which aims to solve this by providing a unified, gym-like environment with well-defined observation and action spaces, facilitating standardized evaluation across diverse benchmarks. We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature and includes AgentLab, a complementary framework that aids in agent creation, testing, and analysis. Our proposed ecosystem offers flexibility for integrating new benchmarks while ensuring consistent evaluation and comprehensive experiment management. As a supporting evidence, we conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks made available in BrowserGym. Among other findings, our results highlight a large discrepancy between OpenAI and Anthropic's latests models, with Claude-3.5-Sonnet leading the way on almost all benchmarks, except on vision-related tasks where GPT-4o is superior. Despite these advancements, our results emphasize that building robust and efficient web agents remains a significant challenge, due to the inherent complexity of real-world web environments and the limitations of current models.","url_abs":"https://arxiv.org/abs/2412.05467v4","url_pdf":"https://arxiv.org/pdf/2412.05467v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-browsergym-ecosystem-for-web-agent","repo_url":"https://github.com/servicenow/agentlab","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"the-browsergym-ecosystem-for-web-agent","repo_url":"https://github.com/servicenow/browsergym","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"the-browsergym-ecosystem-for-web-agent","repo_url":"https://github.com/servicenow/workarena","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"fragmentation","method_name":"Fragmentation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.05467","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.05467"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/servicenow/agentlab","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/servicenow/browsergym","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/servicenow/workarena","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":4,"unverified":20},"by_repo_kind":{"official":{"samples":17,"ran":4,"repositories":2},"listed":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":24,"samples":[{"code_sha256_prefix":"8194ee5c0231da57","entry":"extract_data_items_from_aria","repo":"servicenow/browsergym","repo_kind":"official","path":"browsergym/core/src/browsergym/core/observation.py","file_url":"https://github.com/servicenow/browsergym/blob/HEAD/browsergym/core/src/browsergym/core/observation.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8194ee5c0231da57"}},{"code_sha256_prefix":"b938f1e73d583a18","entry":"get_chatbox_classic","repo":"servicenow/browsergym","repo_kind":"official","path":"browsergym/core/src/browsergym/core/chat.py","file_url":"https://github.com/servicenow/browsergym/blob/HEAD/browsergym/core/src/browsergym/core/chat.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b938f1e73d583a18"}},{"code_sha256_prefix":"e01ad5d24888de64","entry":"get_chatbox_modern","repo":"servicenow/browsergym","repo_kind":"official","path":"browsergym/core/src/browsergym/core/chat.py","file_url":"https://github.com/servicenow/browsergym/blob/HEAD/browsergym/core/src/browsergym/core/chat.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e01ad5d24888de64"}},{"code_sha256_prefix":"b121272b1bd15e8b","entry":"overlay_som","repo":"servicenow/browsergym","repo_kind":"official","path":"browsergym/core/src/browsergym/utils/obs.py","file_url":"https://github.com/servicenow/browsergym/blob/HEAD/browsergym/core/src/browsergym/utils/obs.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b121272b1bd15e8b"}},{"code_sha256_prefix":"6d68d6d5030ccff9","entry":"SNowInstance","repo":"servicenow/workarena","repo_kind":"listed","path":"src/browsergym/workarena/install.py","file_url":"https://github.com/servicenow/workarena/blob/HEAD/src/browsergym/workarena/install.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6d68d6d5030ccff9"}},{"code_sha256_prefix":"877887aa89d5c493","entry":"add_step_timing_to_env_info_decorator","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/benchmarks/abstract_env.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/benchmarks/abstract_env.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"877887aa89d5c493"}},{"code_sha256_prefix":"e0611388db73b85b","entry":"clean_column_names","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/analyze/agent_xray.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/analyze/agent_xray.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e0611388db73b85b"}},{"code_sha256_prefix":"b924a66a3ad86b83","entry":"decrypt_instance_password","repo":"servicenow/workarena","repo_kind":"listed","path":"src/browsergym/workarena/instance.py","file_url":"https://github.com/servicenow/workarena/blob/HEAD/src/browsergym/workarena/instance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b924a66a3ad86b83"}},{"code_sha256_prefix":"86d76ffec286856f","entry":"display_table","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/analyze/agent_xray.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/analyze/agent_xray.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"86d76ffec286856f"}},{"code_sha256_prefix":"45c1db8f1020a60d","entry":"draw_click_indicator","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/agents/agent_utils.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/agents/agent_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"45c1db8f1020a60d"}},{"code_sha256_prefix":"07b31c698d493662","entry":"draw_mouse_pointer","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/agents/agent_utils.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/agents/agent_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"07b31c698d493662"}},{"code_sha256_prefix":"2194414c8a1dfd5d","entry":"encrypt_instance_password","repo":"servicenow/workarena","repo_kind":"listed","path":"src/browsergym/workarena/instance.py","file_url":"https://github.com/servicenow/workarena/blob/HEAD/src/browsergym/workarena/instance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2194414c8a1dfd5d"}},{"code_sha256_prefix":"a9ef891d6b2d4051","entry":"fetch_instances","repo":"servicenow/workarena","repo_kind":"listed","path":"src/browsergym/workarena/instance.py","file_url":"https://github.com/servicenow/workarena/blob/HEAD/src/browsergym/workarena/instance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a9ef891d6b2d4051"}},{"code_sha256_prefix":"15fde3c327650a12","entry":"find_bids_and_xy_pairs","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/analyze/overlay_utils.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/analyze/overlay_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"15fde3c327650a12"}},{"code_sha256_prefix":"567d81b57b0ac04f","entry":"flatten_axtree_to_str","repo":"servicenow/browsergym","repo_kind":"official","path":"browsergym/core/src/browsergym/utils/obs.py","file_url":"https://github.com/servicenow/browsergym/blob/HEAD/browsergym/core/src/browsergym/utils/obs.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"567d81b57b0ac04f"}},{"code_sha256_prefix":"ce62e9936fba1098","entry":"get_categories","repo":"servicenow/workarena","repo_kind":"listed","path":"src/browsergym/workarena/api/category.py","file_url":"https://github.com/servicenow/workarena/blob/HEAD/src/browsergym/workarena/api/category.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ce62e9936fba1098"}},{"code_sha256_prefix":"829a15ad2e0ff974","entry":"get_constants_and_variables","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/analyze/inspect_results.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/analyze/inspect_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"829a15ad2e0ff974"}},{"code_sha256_prefix":"a1798062547f9110","entry":"get_cost_center_sysid","repo":"servicenow/workarena","repo_kind":"listed","path":"src/browsergym/workarena/api/cost_center.py","file_url":"https://github.com/servicenow/workarena/blob/HEAD/src/browsergym/workarena/api/cost_center.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a1798062547f9110"}},{"code_sha256_prefix":"1bf94288772744bd","entry":"is_step_completed","repo":"servicenow/workarena","repo_kind":"listed","path":"src/browsergym/workarena/install.py","file_url":"https://github.com/servicenow/workarena/blob/HEAD/src/browsergym/workarena/install.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1bf94288772744bd"}},{"code_sha256_prefix":"d1c553ebca78057a","entry":"overlay_cross","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/analyze/overlay_utils.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/analyze/overlay_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d1c553ebca78057a"}},{"code_sha256_prefix":"53068f77c8646667","entry":"parse_function_calls","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/analyze/overlay_utils.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/analyze/overlay_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"53068f77c8646667"}},{"code_sha256_prefix":"9b30bcc3faa69153","entry":"pretty_yaml","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/analyze/tapes.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/analyze/tapes.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9b30bcc3faa69153"}},{"code_sha256_prefix":"aa8fc291716bf5f9","entry":"reduce_episodes","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/analyze/inspect_results.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/analyze/inspect_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"aa8fc291716bf5f9"}},{"code_sha256_prefix":"b82aafda2161d7c5","entry":"remove_args_from_col","repo":"servicenow/agentlab","repo_kind":"official","path":"src/agentlab/analyze/agent_xray.py","file_url":"https://github.com/servicenow/agentlab/blob/HEAD/src/agentlab/analyze/agent_xray.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b82aafda2161d7c5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}