{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/guicourse-from-general-vision-language-models","title":"GUICourse: From General Vision Language Models to Versatile GUI Agents","arxiv_id":"2406.11317","date":"2024-06-17","proceeding":null,"authors":["Wentong Chen","Junbo Cui","Jinyi Hu","Yujia Qin","Junjie Fang","Yue Zhao","Chongyi Wang","Jun Liu","Guirong Chen","Yupeng Huo","Yuan YAO","Yankai Lin","Zhiyuan Liu","Maosong Sun"],"abstract":"Utilizing Graphic User Interface (GUI) for human-computer interaction is essential for accessing a wide range of digital tools. Recent advancements in Vision Language Models (VLMs) highlight the compelling potential to develop versatile agents to help humans finish GUI navigation tasks. However, current VLMs are challenged in terms of fundamental abilities (OCR and grounding) and GUI knowledge (the functions and control methods of GUI elements), preventing them from becoming practical GUI agents. To solve these challenges, we contribute GUICourse, a suite of datasets to train visual-based GUI agents from general VLMs. First, we introduce the GUIEnv dataset to strengthen the OCR and grounding capabilities of VLMs. Then, we introduce the GUIAct and GUIChat datasets to enrich their knowledge of GUI components and interactions. Experiments demonstrate that our GUI agents have better performance on common GUI tasks than their baseline VLMs. Even the small-size GUI agent (with 3.1B parameters) can still work well on single-step and multi-step GUI tasks. Finally, we analyze the different varieties in the training stage of this agent by ablation study. Our source codes and datasets are released at https://github.com/yiye3/GUICourse.","url_abs":"https://arxiv.org/abs/2406.11317v2","url_pdf":"https://arxiv.org/pdf/2406.11317v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"guicourse-from-general-vision-language-models","repo_url":"https://github.com/yiye3/guicourse","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"natural-language-visual-grounding","task_name":"Natural Language Visual Grounding"},{"task_slug":"optical-character-recognition","task_name":"Optical Character Recognition (OCR)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/natural-language-visual-grounding-on","task":"Natural Language Visual Grounding","dataset":"ScreenSpot","model":"Qwen-GUI","rank_in_archive_order":15,"of":18,"metrics":{"Accuracy (%)":"28.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.11317","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.11317"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yiye3/guicourse","reach":{"status":"ok"}}],"summary":{"ran":14,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":16,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"30255d9ba1ef6094","entry":"convert_guiact_to_qwen_format","repo":"yiye3/guicourse","repo_kind":"official","path":"data_preprocess/merge_data.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/data_preprocess/merge_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30255d9ba1ef6094"}},{"code_sha256_prefix":"a8cc5e1c280fda6a","entry":"convert_guienv_to_qwen_format","repo":"yiye3/guicourse","repo_kind":"official","path":"data_preprocess/merge_data.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/data_preprocess/merge_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8cc5e1c280fda6a"}},{"code_sha256_prefix":"ecc04078e2043893","entry":"decode_base64_to_image","repo":"yiye3/guicourse","repo_kind":"official","path":"data_load.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/data_load.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ecc04078e2043893"}},{"code_sha256_prefix":"335b6330b9af2164","entry":"eval_bbox2text","repo":"yiye3/guicourse","repo_kind":"official","path":"evaluation/eval_single_action.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/evaluation/eval_single_action.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"335b6330b9af2164"}},{"code_sha256_prefix":"f068f39a25e1a842","entry":"infer","repo":"yiye3/guicourse","repo_kind":"official","path":"Qwen-SFT&Infer/infer.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/Qwen-SFT%26Infer/infer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f068f39a25e1a842"}},{"code_sha256_prefix":"036a213486b2f4b4","entry":"iou","repo":"yiye3/guicourse","repo_kind":"official","path":"utils.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"036a213486b2f4b4"}},{"code_sha256_prefix":"d2d934fd1a05b9e1","entry":"is_pass_check","repo":"yiye3/guicourse","repo_kind":"official","path":"data_preprocess/convert_to_sft_instructions.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/data_preprocess/convert_to_sft_instructions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2d934fd1a05b9e1"}},{"code_sha256_prefix":"6ec4ec7ae41b4345","entry":"load_data","repo":"yiye3/guicourse","repo_kind":"official","path":"Qwen-SFT&Infer/infer.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/Qwen-SFT%26Infer/infer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ec4ec7ae41b4345"}},{"code_sha256_prefix":"baaebe9658032c54","entry":"load_model_and_tokenizer","repo":"yiye3/guicourse","repo_kind":"official","path":"Qwen-SFT&Infer/infer.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/Qwen-SFT%26Infer/infer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"baaebe9658032c54"}},{"code_sha256_prefix":"4b39866ea5e2b70d","entry":"parse_box","repo":"yiye3/guicourse","repo_kind":"official","path":"utils.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b39866ea5e2b70d"}},{"code_sha256_prefix":"cb7585c0b75f73f9","entry":"parse_point","repo":"yiye3/guicourse","repo_kind":"official","path":"utils.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb7585c0b75f73f9"}},{"code_sha256_prefix":"8b733c9fc590e4c1","entry":"preprocess","repo":"yiye3/guicourse","repo_kind":"official","path":"Qwen-SFT&Infer/finetune.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/Qwen-SFT%26Infer/finetune.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b733c9fc590e4c1"}},{"code_sha256_prefix":"aedd559eaebb8985","entry":"read_data","repo":"yiye3/guicourse","repo_kind":"official","path":"data_preprocess/convert_parquet_to_png.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/data_preprocess/convert_parquet_to_png.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aedd559eaebb8985"}},{"code_sha256_prefix":"61ec4f7400ef2dac","entry":"read_json","repo":"yiye3/guicourse","repo_kind":"official","path":"data_load.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/data_load.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61ec4f7400ef2dac"}},{"code_sha256_prefix":"4906b1a9823dc7d6","entry":"read_parquet","repo":"yiye3/guicourse","repo_kind":"official","path":"data_load.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/data_load.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4906b1a9823dc7d6"}},{"code_sha256_prefix":"25de1b1cb65b6b3c","entry":"elements_visual","repo":"yiye3/guicourse","repo_kind":"official","path":"data_visualization.py","file_url":"https://github.com/yiye3/guicourse/blob/HEAD/data_visualization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"25de1b1cb65b6b3c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}