{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/showui-one-vision-language-action-model-for","title":"ShowUI: One Vision-Language-Action Model for GUI Visual Agent","arxiv_id":"2411.17465","date":"2024-11-26","proceeding":"CVPR 2025 1","authors":["Kevin Qinghong Lin","Linjie Li","Difei Gao","Zhengyuan Yang","Shiwei Wu","Zechen Bai","Weixian Lei","Lijuan Wang","Mike Zheng Shou"],"abstract":"Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. While most agents are language-based, relying on closed-source API with text-rich meta-information (e.g., HTML or accessibility tree), they show limitations in perceiving UI visuals as humans do, highlighting the need for GUI visual agents. In this work, we develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations: (i) UI-Guided Visual Token Selection to reduce computational costs by formulating screenshots as an UI connected graph, adaptively identifying their redundant relationship and serve as the criteria for token selection during self-attention blocks; (ii) Interleaved Vision-Language-Action Streaming that flexibly unifies diverse needs within GUI tasks, enabling effective management of visual-action history in navigation or pairing multi-turn query-action sequences per screenshot to enhance training efficiency; (iii) Small-scale High-quality GUI Instruction-following Datasets by careful data curation and employing a resampling strategy to address significant data type imbalances. With above components, ShowUI, a lightweight 2B model using 256K data, achieves a strong 75.1% accuracy in zero-shot screenshot grounding. Its UI-guided token selection further reduces 33% of redundant visual tokens during training and speeds up the performance by 1.4x. Navigation experiments across web Mind2Web, mobile AITW, and online MiniWob environments further underscore the effectiveness and potential of our model in advancing GUI visual agents. The models are available at https://github.com/showlab/ShowUI.","url_abs":"https://arxiv.org/abs/2411.17465v1","url_pdf":"https://arxiv.org/pdf/2411.17465v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"showui-one-vision-language-action-model-for","repo_url":"https://github.com/showlab/showui","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"natural-language-visual-grounding","task_name":"Natural Language Visual Grounding"},{"task_slug":"vision-language-action","task_name":"Vision-Language-Action"}],"methods":[{"method_slug":"1-bit-adam","method_name":"1-bit Adam"},{"method_slug":"adam","method_name":"Adam"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/natural-language-visual-grounding-on","task":"Natural Language Visual Grounding","dataset":"ScreenSpot","model":"ShowUI","rank_in_archive_order":7,"of":18,"metrics":{"Accuracy (%)":"75.1"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-visual-grounding-on","task":"Natural Language Visual Grounding","dataset":"ScreenSpot","model":"ShowUI-G","rank_in_archive_order":8,"of":18,"metrics":{"Accuracy (%)":"75.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.17465","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.17465"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/showlab/showui","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"unverified":15},"by_repo_kind":{"official":{"samples":16,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"showlab/showui","repo_kind":"official","path":"model/showui/modeling_showui.py","file_url":"https://github.com/showlab/showui/blob/HEAD/model/showui/modeling_showui.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"eb4d16c3bba7cff8","entry":"apply_multimodal_rotary_pos_emb","repo":"showlab/showui","repo_kind":"official","path":"model/showui/modeling_showui.py","file_url":"https://github.com/showlab/showui/blob/HEAD/model/showui/modeling_showui.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eb4d16c3bba7cff8"}},{"code_sha256_prefix":"48f12d3a9c4a15e8","entry":"apply_rotary_pos_emb_vision","repo":"showlab/showui","repo_kind":"official","path":"model/showui/modeling_showui.py","file_url":"https://github.com/showlab/showui/blob/HEAD/model/showui/modeling_showui.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"48f12d3a9c4a15e8"}},{"code_sha256_prefix":"6bc574a30ca765af","entry":"broadcast_value","repo":"showlab/showui","repo_kind":"official","path":"main/eval_aitw.py","file_url":"https://github.com/showlab/showui/blob/HEAD/main/eval_aitw.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6bc574a30ca765af"}},{"code_sha256_prefix":"fdf1a1edeca8987f","entry":"calculate_aitw_metrics","repo":"showlab/showui","repo_kind":"official","path":"main/eval_aitw.py","file_url":"https://github.com/showlab/showui/blob/HEAD/main/eval_aitw.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fdf1a1edeca8987f"}},{"code_sha256_prefix":"c3c22ae25acb9f88","entry":"calculate_f1","repo":"showlab/showui","repo_kind":"official","path":"main/eval_mind2web.py","file_url":"https://github.com/showlab/showui/blob/HEAD/main/eval_mind2web.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c3c22ae25acb9f88"}},{"code_sha256_prefix":"00c1146afbae58e2","entry":"find_target_linear_names","repo":"showlab/showui","repo_kind":"official","path":"merge_weight.py","file_url":"https://github.com/showlab/showui/blob/HEAD/merge_weight.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"00c1146afbae58e2"}},{"code_sha256_prefix":"954a7534eb56091e","entry":"find_target_linear_names","repo":"showlab/showui","repo_kind":"official","path":"model/utils.py","file_url":"https://github.com/showlab/showui/blob/HEAD/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"954a7534eb56091e"}},{"code_sha256_prefix":"722a5095911699c3","entry":"get_bbox","repo":"showlab/showui","repo_kind":"official","path":"main/eval_mind2web.py","file_url":"https://github.com/showlab/showui/blob/HEAD/main/eval_mind2web.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"722a5095911699c3"}},{"code_sha256_prefix":"b6b2e338236414ad","entry":"get_bbox","repo":"showlab/showui","repo_kind":"official","path":"main/eval_screenspot.py","file_url":"https://github.com/showlab/showui/blob/HEAD/main/eval_screenspot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b6b2e338236414ad"}},{"code_sha256_prefix":"be7b764c7115cbb6","entry":"get_select_mask","repo":"showlab/showui","repo_kind":"official","path":"model/showui/utils.py","file_url":"https://github.com/showlab/showui/blob/HEAD/model/showui/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be7b764c7115cbb6"}},{"code_sha256_prefix":"9dab4813f6ad1ea8","entry":"load_sharded_weights","repo":"showlab/showui","repo_kind":"official","path":"merge_weight.py","file_url":"https://github.com/showlab/showui/blob/HEAD/merge_weight.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9dab4813f6ad1ea8"}},{"code_sha256_prefix":"331eaaacd445f7a8","entry":"parse_args","repo":"showlab/showui","repo_kind":"official","path":"merge_weight.py","file_url":"https://github.com/showlab/showui/blob/HEAD/merge_weight.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"331eaaacd445f7a8"}},{"code_sha256_prefix":"46b7e6657cf825e7","entry":"parse_layer_type","repo":"showlab/showui","repo_kind":"official","path":"model/utils.py","file_url":"https://github.com/showlab/showui/blob/HEAD/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46b7e6657cf825e7"}},{"code_sha256_prefix":"cfb0ea42d6d1e0ca","entry":"pointinbbox","repo":"showlab/showui","repo_kind":"official","path":"main/eval_screenspot.py","file_url":"https://github.com/showlab/showui/blob/HEAD/main/eval_screenspot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cfb0ea42d6d1e0ca"}},{"code_sha256_prefix":"a50444314d796645","entry":"smart_resize","repo":"showlab/showui","repo_kind":"official","path":"model/showui/image_processing_showui.py","file_url":"https://github.com/showlab/showui/blob/HEAD/model/showui/image_processing_showui.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a50444314d796645"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}