{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mp-gui-modality-perception-with-mllms-for-gui","title":"MP-GUI: Modality Perception with MLLMs for GUI Understanding","arxiv_id":"2503.14021","date":"2025-03-18","proceeding":"CVPR 2025 1","authors":["Ziwei Wang","Weizhi Chen","Leyang Yang","Sheng Zhou","Shengchu Zhao","Hanbei Zhan","Jiongchao Jin","Liangcheng Li","Zirui Shao","Jiajun Bu"],"abstract":"Graphical user interface (GUI) has become integral to modern society, making it crucial to be understood for human-centric systems. However, unlike natural images or documents, GUIs comprise artificially designed graphical elements arranged to convey specific semantic meanings. Current multi-modal large language models (MLLMs) already proficient in processing graphical and textual components suffer from hurdles in GUI understanding due to the lack of explicit spatial structure modeling. Moreover, obtaining high-quality spatial structure data is challenging due to privacy issues and noisy environments. To address these challenges, we present MP-GUI, a specially designed MLLM for GUI understanding. MP-GUI features three precisely specialized perceivers to extract graphical, textual, and spatial modalities from the screen as GUI-tailored visual clues, with spatial structure refinement strategy and adaptively combined via a fusion gate to meet the specific preferences of different GUI understanding tasks. To cope with the scarcity of training data, we also introduce a pipeline for automatically data collecting. Extensive experiments demonstrate that MP-GUI achieves impressive results on various GUI understanding tasks with limited data.","url_abs":"https://arxiv.org/abs/2503.14021v1","url_pdf":"https://arxiv.org/pdf/2503.14021v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mp-gui-modality-perception-with-mllms-for-gui","repo_url":"https://github.com/BigTaige/MP-GUI","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.14021","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.14021"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/BigTaige/MP-GUI","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"unverified":11},"by_repo_kind":{"official":{"samples":12,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"10893c4608c08075","entry":"split_to_even_chunks","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"model/internvl/patch/train_sampler_patch.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/model/internvl/patch/train_sampler_patch.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10893c4608c08075"}},{"code_sha256_prefix":"3408dd47e939d760","entry":"apply_rotary_pos_emb","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"model/internvl/patch/llama2_flash_attn_monkey_patch.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/model/internvl/patch/llama2_flash_attn_monkey_patch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3408dd47e939d760"}},{"code_sha256_prefix":"dd863a03e032ee31","entry":"chat","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"pipeline/vllm_pipeline_v2.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/pipeline/vllm_pipeline_v2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd863a03e032ee31"}},{"code_sha256_prefix":"c025eeea3e7ee863","entry":"concat_pad_data_collator","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"model/internvl/patch/pad_data_collator.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/model/internvl/patch/pad_data_collator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c025eeea3e7ee863"}},{"code_sha256_prefix":"021dddfe7e4fc462","entry":"forward","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"model/internvl/patch/llama_flash_attn_monkey_patch.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/model/internvl/patch/llama_flash_attn_monkey_patch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"021dddfe7e4fc462"}},{"code_sha256_prefix":"f67aecf10b83d6b9","entry":"forward_2","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"model/internvl/patch/llama_flash_attn_monkey_patch.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/model/internvl/patch/llama_flash_attn_monkey_patch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f67aecf10b83d6b9"}},{"code_sha256_prefix":"e24dcde1d77081a5","entry":"get_clean_data","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"data_tools/XML_format_get_spatial_relationship_prediction_data.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/data_tools/XML_format_get_spatial_relationship_prediction_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e24dcde1d77081a5"}},{"code_sha256_prefix":"04b172bb0c32a0de","entry":"get_conv_template","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"model/internvl/conversation.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/model/internvl/conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04b172bb0c32a0de"}},{"code_sha256_prefix":"fac53f6bae1b69a4","entry":"get_length_grouped_indices","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"model/internvl/patch/train_sampler_patch.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/model/internvl/patch/train_sampler_patch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac53f6bae1b69a4"}},{"code_sha256_prefix":"63bbe0118077916d","entry":"judge","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"data_tools/XML_format_get_spatial_relationship_prediction_data.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/data_tools/XML_format_get_spatial_relationship_prediction_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63bbe0118077916d"}},{"code_sha256_prefix":"0001daa41fec04a5","entry":"label_formulate","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"data_tools/XML_format_get_spatial_relationship_prediction_data.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/data_tools/XML_format_get_spatial_relationship_prediction_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0001daa41fec04a5"}},{"code_sha256_prefix":"c98153c6978bb741","entry":"pad_data_collator","repo":"BigTaige/MP-GUI","repo_kind":"official","path":"model/internvl/patch/pad_data_collator.py","file_url":"https://github.com/BigTaige/MP-GUI/blob/HEAD/model/internvl/patch/pad_data_collator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c98153c6978bb741"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}