{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visualwebarena-evaluating-multimodal-agents","title":"VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks","arxiv_id":"2401.13649","date":"2024-01-24","proceeding":null,"authors":["Jing Yu Koh","Robert Lo","Lawrence Jang","Vikram Duvvur","Ming Chong Lim","Po-Yu Huang","Graham Neubig","Shuyan Zhou","Ruslan Salakhutdinov","Daniel Fried"],"abstract":"Autonomous agents capable of planning, reasoning, and executing actions on the web offer a promising avenue for automating computer tasks. However, the majority of existing benchmarks primarily focus on text-based agents, neglecting many natural tasks that require visual information to effectively solve. Given that most computer interfaces cater to human perception, visual information often augments textual data in ways that text-only models struggle to harness effectively. To bridge this gap, we introduce VisualWebArena, a benchmark designed to assess the performance of multimodal web agents on realistic \\textit{visually grounded tasks}. VisualWebArena comprises of a set of diverse and complex web-based tasks that evaluate various capabilities of autonomous multimodal agents. To perform on this benchmark, agents need to accurately process image-text inputs, interpret natural language instructions, and execute actions on websites to accomplish user-defined objectives. We conduct an extensive evaluation of state-of-the-art LLM-based autonomous agents, including several multimodal models. Through extensive quantitative and qualitative analysis, we identify several limitations of text-only LLM agents, and reveal gaps in the capabilities of state-of-the-art multimodal language agents. VisualWebArena provides a framework for evaluating multimodal autonomous language agents, and offers insights towards building stronger autonomous agents for the web. Our code, baseline models, and data is publicly available at https://jykoh.com/vwa.","url_abs":"https://arxiv.org/abs/2401.13649v2","url_pdf":"https://arxiv.org/pdf/2401.13649v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visualwebarena-evaluating-multimodal-agents","repo_url":"https://github.com/web-arena-x/visualwebarena","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2401.13649","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.13649"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/web-arena-x/visualwebarena","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"32bc38b0f6c79438","entry":"action2str","repo":"web-arena-x/visualwebarena","repo_kind":"official","path":"browser_env/actions.py","file_url":"https://github.com/web-arena-x/visualwebarena/blob/HEAD/browser_env/actions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32bc38b0f6c79438"}},{"code_sha256_prefix":"824ae078c5d3295a","entry":"get_site_comb_from_filepath","repo":"web-arena-x/visualwebarena","repo_kind":"official","path":"browser_env/auto_login.py","file_url":"https://github.com/web-arena-x/visualwebarena/blob/HEAD/browser_env/auto_login.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"824ae078c5d3295a"}},{"code_sha256_prefix":"1829c408a278c974","entry":"parse_action","repo":"web-arena-x/visualwebarena","repo_kind":"official","path":"browser_env/envs.py","file_url":"https://github.com/web-arena-x/visualwebarena/blob/HEAD/browser_env/envs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1829c408a278c974"}},{"code_sha256_prefix":"d8254925e6697a59","entry":"pil_to_b64","repo":"web-arena-x/visualwebarena","repo_kind":"official","path":"browser_env/utils.py","file_url":"https://github.com/web-arena-x/visualwebarena/blob/HEAD/browser_env/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8254925e6697a59"}},{"code_sha256_prefix":"71ec370bd2568e5b","entry":"pil_to_vertex","repo":"web-arena-x/visualwebarena","repo_kind":"official","path":"browser_env/utils.py","file_url":"https://github.com/web-arena-x/visualwebarena/blob/HEAD/browser_env/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71ec370bd2568e5b"}},{"code_sha256_prefix":"04ba3dcf1bb13e4f","entry":"png_bytes_to_numpy","repo":"web-arena-x/visualwebarena","repo_kind":"official","path":"browser_env/utils.py","file_url":"https://github.com/web-arena-x/visualwebarena/blob/HEAD/browser_env/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04ba3dcf1bb13e4f"}},{"code_sha256_prefix":"b3cfdfe328f76c2d","entry":"remove_unicode","repo":"web-arena-x/visualwebarena","repo_kind":"official","path":"browser_env/processors.py","file_url":"https://github.com/web-arena-x/visualwebarena/blob/HEAD/browser_env/processors.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3cfdfe328f76c2d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}