{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-24348","title":"OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents","arxiv_id":"2604.24348","date":"2026-04-27","proceeding":null,"authors":["Zheng Wu","Yi Hua","Zhaoyuan Huang","Chenhao Xue","Yijie Lu","Pengzhou Cheng","Zongru Wu","Lingzhong Dong","Gongshen Liu","Xinghao Jiang","Zhuosheng Zhang"],"abstract":"The evolution of Multimodal Large Language Models (MLLMs) has shifted the focus from text generation to active behavioral execution, particularly via OS agents navigating complex GUIs. However, the transition of these agents into trustworthy daily partners is hindered by a lack of rigorous evaluation regarding safety, efficiency, and multi-modal robustness. Current benchmarks suffer from narrow safety scenarios, noisy trajectory labeling, and limited robustness metrics. To bridge this gap, we propose OS-SPEAR, a comprehensive toolkit for the systematic analysis of OS agents across four dimensions: Safety, Performance, Efficiency, and Robustness. OS-SPEAR introduces four specialized subsets: (1) a S(afety)-subset encompassing diverse environment- and human-induced hazards; (2) a P(erformance)-subset curated via trajectory value estimation and stratified sampling; (3) an E(fficiency)-subset quantifying performance through the dual lenses of temporal latency and token consumption; and (4) a R(obustness)-subset that applies cross-modal disturbances to both visual and textual inputs. Additionally, we provide an automated analysis tool to generate human-readable diagnostic reports. We conduct an extensive evaluation of 22 popular OS agents using OS-SPEAR. Our empirical results reveal critical insights into the current landscape: notably, a prevalent trade-off between efficiency and safety or robustness, the performance superiority of specialized agents over general-purpose models, and varying robustness vulnerabilities across different modalities. By providing a multidimensional ranking and a standardized evaluation framework, OS-SPEAR offers a foundational resource for developing the next generation of reliable and efficient OS agents. The dataset and codes are available at https://github.com/Wuzheng02/OS-SPEAR.","url_abs":"https://arxiv.org/abs/2604.24348","url_pdf":"https://arxiv.org/pdf/2604.24348","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.24348","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.24348"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Wuzheng02/OS-SPEAR","reach":null}],"summary":{"ran_honours":1,"ran":9},"by_repo_kind":{"found_in_text":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"6ab14682c87e7bce","entry":"calculate_f1_score","repo":"Wuzheng02/OS-SPEAR","repo_kind":"found_in_text","path":"eval/eval_single.py","file_url":"https://github.com/Wuzheng02/OS-SPEAR/blob/HEAD/eval/eval_single.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ab14682c87e7bce"}},{"code_sha256_prefix":"6a1b8329e5eb58f2","entry":"extract_point","repo":"Wuzheng02/OS-SPEAR","repo_kind":"found_in_text","path":"eval/eval_single.py","file_url":"https://github.com/Wuzheng02/OS-SPEAR/blob/HEAD/eval/eval_single.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a1b8329e5eb58f2"}},{"code_sha256_prefix":"182ea30fe35445e7","entry":"parse_action_type","repo":"Wuzheng02/OS-SPEAR","repo_kind":"found_in_text","path":"eval/eval_single.py","file_url":"https://github.com/Wuzheng02/OS-SPEAR/blob/HEAD/eval/eval_single.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"182ea30fe35445e7"}},{"code_sha256_prefix":"2558a78be5cf41fc","entry":"parse_txt_log","repo":"Wuzheng02/OS-SPEAR","repo_kind":"found_in_text","path":"construct/P_subset.py","file_url":"https://github.com/Wuzheng02/OS-SPEAR/blob/HEAD/construct/P_subset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2558a78be5cf41fc"}},{"code_sha256_prefix":"4c644a9a6cf99165","entry":"read_text","repo":"Wuzheng02/OS-SPEAR","repo_kind":"found_in_text","path":"report/evaluate.py","file_url":"https://github.com/Wuzheng02/OS-SPEAR/blob/HEAD/report/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c644a9a6cf99165"}},{"code_sha256_prefix":"d2545774aed5c26f","entry":"safe_drop","repo":"Wuzheng02/OS-SPEAR","repo_kind":"found_in_text","path":"report/evaluate.py","file_url":"https://github.com/Wuzheng02/OS-SPEAR/blob/HEAD/report/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2545774aed5c26f"}},{"code_sha256_prefix":"32350ee94c299bb0","entry":"safe_float","repo":"Wuzheng02/OS-SPEAR","repo_kind":"found_in_text","path":"report/evaluate.py","file_url":"https://github.com/Wuzheng02/OS-SPEAR/blob/HEAD/report/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32350ee94c299bb0"}},{"code_sha256_prefix":"1f7ef3ea8b87966c","entry":"transfer_qwen3vl2atlas","repo":"Wuzheng02/OS-SPEAR","repo_kind":"found_in_text","path":"eval/transfer.py","file_url":"https://github.com/Wuzheng02/OS-SPEAR/blob/HEAD/eval/transfer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1f7ef3ea8b87966c"}},{"code_sha256_prefix":"ed306c09be100acc","entry":"transfer_tars15toatlas","repo":"Wuzheng02/OS-SPEAR","repo_kind":"found_in_text","path":"eval/transfer.py","file_url":"https://github.com/Wuzheng02/OS-SPEAR/blob/HEAD/eval/transfer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ed306c09be100acc"}},{"code_sha256_prefix":"162e08ed7676f726","entry":"transfer_tars2atlas","repo":"Wuzheng02/OS-SPEAR","repo_kind":"found_in_text","path":"eval/transfer.py","file_url":"https://github.com/Wuzheng02/OS-SPEAR/blob/HEAD/eval/transfer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"162e08ed7676f726"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}