{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/windows-agent-arena-evaluating-multi-modal-os","title":"Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale","arxiv_id":"2409.08264","date":"2024-09-12","proceeding":null,"authors":["Rogerio Bonatti","Dan Zhao","Francesco Bonacci","Dillon Dupont","Sara Abdali","Yinheng Li","Yadong Lu","Justin Wagle","Kazuhito Koishida","Arthur Bucker","Lawrence Jang","Zack Hui"],"abstract":"Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena. Webpage: https://microsoft.github.io/WindowsAgentArena Code: https://github.com/microsoft/WindowsAgentArena","url_abs":"https://arxiv.org/abs/2409.08264v2","url_pdf":"https://arxiv.org/pdf/2409.08264v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"windows-agent-arena-evaluating-multi-modal-os","repo_url":"https://github.com/microsoft/windowsagentarena","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.08264","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.08264"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/windowsagentarena","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e95e71722d8ee6d7","entry":"get_directory_structure","repo":"microsoft/windowsagentarena","repo_kind":"official","path":"src/win-arena-container/client/evaluation_examples_windows/create_json.py","file_url":"https://github.com/microsoft/windowsagentarena/blob/HEAD/src/win-arena-container/client/evaluation_examples_windows/create_json.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e95e71722d8ee6d7"}},{"code_sha256_prefix":"c84f9385b9ff870c","entry":"get_result","repo":"microsoft/windowsagentarena","repo_kind":"official","path":"src/win-arena-container/client/show_result.py","file_url":"https://github.com/microsoft/windowsagentarena/blob/HEAD/src/win-arena-container/client/show_result.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c84f9385b9ff870c"}},{"code_sha256_prefix":"a3a1e6059333e000","entry":"is_leaf_node","repo":"microsoft/windowsagentarena","repo_kind":"official","path":"src/win-arena-container/client/mm_agents/navi/a11y_demo.py","file_url":"https://github.com/microsoft/windowsagentarena/blob/HEAD/src/win-arena-container/client/mm_agents/navi/a11y_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3a1e6059333e000"}},{"code_sha256_prefix":"2a014475a5e7301b","entry":"prev_actions_to_string","repo":"microsoft/windowsagentarena","repo_kind":"official","path":"src/win-arena-container/client/mm_agents/navi/agent.py","file_url":"https://github.com/microsoft/windowsagentarena/blob/HEAD/src/win-arena-container/client/mm_agents/navi/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a014475a5e7301b"}},{"code_sha256_prefix":"06c42a797d1bb70c","entry":"remove_min_leading_spaces","repo":"microsoft/windowsagentarena","repo_kind":"official","path":"src/win-arena-container/client/mm_agents/navi/agent.py","file_url":"https://github.com/microsoft/windowsagentarena/blob/HEAD/src/win-arena-container/client/mm_agents/navi/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06c42a797d1bb70c"}},{"code_sha256_prefix":"7aa29e3ad527ce33","entry":"resize_image_openai","repo":"microsoft/windowsagentarena","repo_kind":"official","path":"src/win-arena-container/client/mm_agents/navi/agent.py","file_url":"https://github.com/microsoft/windowsagentarena/blob/HEAD/src/win-arena-container/client/mm_agents/navi/agent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7aa29e3ad527ce33"}},{"code_sha256_prefix":"1e0a1c4c6aeb27c3","entry":"get_a11y","repo":"microsoft/windowsagentarena","repo_kind":"official","path":"src/win-arena-container/client/mm_agents/navi/a11y_demo.py","file_url":"https://github.com/microsoft/windowsagentarena/blob/HEAD/src/win-arena-container/client/mm_agents/navi/a11y_demo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e0a1c4c6aeb27c3"}},{"code_sha256_prefix":"42369f375e5d6c37","entry":"get_screen","repo":"microsoft/windowsagentarena","repo_kind":"official","path":"src/win-arena-container/client/mm_agents/navi/a11y_demo.py","file_url":"https://github.com/microsoft/windowsagentarena/blob/HEAD/src/win-arena-container/client/mm_agents/navi/a11y_demo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42369f375e5d6c37"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}