{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cogagent-a-visual-language-model-for-gui","title":"CogAgent: A Visual Language Model for GUI Agents","arxiv_id":"2312.08914","date":"2023-12-14","proceeding":"CVPR 2024 1","authors":["Wenyi Hong","Weihan Wang","Qingsong Lv","Jiazheng Xu","Wenmeng Yu","Junhui Ji","Yan Wang","Zihan Wang","Yuxuan Zhang","Juanzi Li","Bin Xu","Yuxiao Dong","Ming Ding","Jie Tang"],"abstract":"People are spending an enormous amount of time on digital devices through graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large language models (LLMs) such as ChatGPT can assist people in tasks like writing emails, but struggle to understand and interact with GUIs, thus limiting their potential to increase automation levels. In this paper, we introduce CogAgent, an 18-billion-parameter visual language model (VLM) specializing in GUI understanding and navigation. By utilizing both low-resolution and high-resolution image encoders, CogAgent supports input at a resolution of 1120*1120, enabling it to recognize tiny page elements and text. As a generalist visual language model, CogAgent achieves the state of the art on five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA, Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using only screenshots as input, outperforms LLM-based methods that consume extracted HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW, advancing the state of the art. The model and codes are available at https://github.com/THUDM/CogVLM, with a new version of CogAgent-9B-20241220 available at https://github.com/THUDM/CogAgent.","url_abs":"https://arxiv.org/abs/2312.08914v3","url_pdf":"https://arxiv.org/pdf/2312.08914v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cogagent-a-visual-language-model-for-gui","repo_url":"https://github.com/THUDM/CogAgent","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"cogagent-a-visual-language-model-for-gui","repo_url":"https://github.com/thudm/cogvlm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"cogagent-a-visual-language-model-for-gui","repo_url":"https://github.com/digirl-agent/digirl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":null,"task_name":"MM-Vet"},{"task_slug":"natural-language-visual-grounding","task_name":"Natural Language Visual Grounding"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/on-1","task":"","dataset":"","model":"","rank_in_archive_order":4,"of":6,"metrics":{"":"-3.5"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-visual-grounding-on","task":"Natural Language Visual Grounding","dataset":"ScreenSpot","model":"CogAgent","rank_in_archive_order":13,"of":18,"metrics":{"Accuracy (%)":"47.4"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"CogAgent","rank_in_archive_order":52,"of":231,"metrics":{"GPT-4 score":"52.8","Params":"18B"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-mm-vet-v2","task":"Visual Question Answering","dataset":"MM-Vet v2","model":"CogAgent-Chat","rank_in_archive_order":20,"of":24,"metrics":{"GPT-4 score":"34.7±0.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.08914","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.08914"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/THUDM/CogAgent","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/digirl-agent/digirl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thudm/cogvlm","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":12,"unverified":6},"by_repo_kind":{"official":{"samples":18,"ran":12,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"54a2b4c8804d95b2","entry":"data_collator","repo":"thudm/cogvlm","repo_kind":"official","path":"finetune_demo/evaluate_cogvlm_demo.py","file_url":"https://github.com/thudm/cogvlm/blob/HEAD/finetune_demo/evaluate_cogvlm_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54a2b4c8804d95b2"}},{"code_sha256_prefix":"afd5a3b7c43f083d","entry":"data_collator","repo":"thudm/cogvlm","repo_kind":"official","path":"finetune_demo/evaluate_cogagent_demo.py","file_url":"https://github.com/thudm/cogvlm/blob/HEAD/finetune_demo/evaluate_cogagent_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"afd5a3b7c43f083d"}},{"code_sha256_prefix":"1f0bc3f4321973e5","entry":"encode_image","repo":"THUDM/CogAgent","repo_kind":"official","path":"app/client.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/app/client.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f0bc3f4321973e5"}},{"code_sha256_prefix":"f3df71c28fae6837","entry":"override_dist_dtype_device_args","repo":"thudm/cogvlm","repo_kind":"official","path":"utils/models/cogvlm_model.py","file_url":"https://github.com/thudm/cogvlm/blob/HEAD/utils/models/cogvlm_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f3df71c28fae6837"}},{"code_sha256_prefix":"d8de666d9830ab8d","entry":"postprocess_image","repo":"thudm/cogvlm","repo_kind":"official","path":"composite_demo/conversation.py","file_url":"https://github.com/thudm/cogvlm/blob/HEAD/composite_demo/conversation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d8de666d9830ab8d"}},{"code_sha256_prefix":"2ce3e067b29fa052","entry":"postprocess_text","repo":"thudm/cogvlm","repo_kind":"official","path":"composite_demo/conversation.py","file_url":"https://github.com/thudm/cogvlm/blob/HEAD/composite_demo/conversation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2ce3e067b29fa052"}},{"code_sha256_prefix":"363f5a1eb1b85d3b","entry":"preprocess_messages","repo":"THUDM/CogAgent","repo_kind":"official","path":"inference/web_demo.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/inference/web_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"363f5a1eb1b85d3b"}},{"code_sha256_prefix":"98595f749058567a","entry":"process_history_and_images","repo":"THUDM/CogAgent","repo_kind":"official","path":"app/openai_demo.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/app/openai_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98595f749058567a"}},{"code_sha256_prefix":"e83d1a79f13a1746","entry":"process_history_and_images","repo":"THUDM/CogAgent","repo_kind":"official","path":"app/vllm_openai_server.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/app/vllm_openai_server.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e83d1a79f13a1746"}},{"code_sha256_prefix":"dc039cc4ef751f3a","entry":"process_image_without_resize","repo":"thudm/cogvlm","repo_kind":"official","path":"basic_demo/web_demo.py","file_url":"https://github.com/thudm/cogvlm/blob/HEAD/basic_demo/web_demo.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dc039cc4ef751f3a"}},{"code_sha256_prefix":"c43bbdf5c24a61ab","entry":"undo_last_round","repo":"THUDM/CogAgent","repo_kind":"official","path":"inference/web_demo.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/inference/web_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c43bbdf5c24a61ab"}},{"code_sha256_prefix":"67160abc4bee8174","entry":"user","repo":"THUDM/CogAgent","repo_kind":"official","path":"inference/web_demo.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/inference/web_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67160abc4bee8174"}},{"code_sha256_prefix":"21b986b1859a8a51","entry":"create_chat_completion","repo":"THUDM/CogAgent","repo_kind":"official","path":"app/client.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/app/client.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"21b986b1859a8a51"}},{"code_sha256_prefix":"7728f4b3ad854aea","entry":"formatting_input","repo":"THUDM/CogAgent","repo_kind":"official","path":"app/client.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/app/client.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7728f4b3ad854aea"}},{"code_sha256_prefix":"34b717a187133b8a","entry":"generate_cogagent","repo":"THUDM/CogAgent","repo_kind":"official","path":"app/openai_demo.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/app/openai_demo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34b717a187133b8a"}},{"code_sha256_prefix":"459bb9214094ea8f","entry":"load_tokenizer_and_model","repo":"THUDM/CogAgent","repo_kind":"official","path":"finetune/finetune.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/finetune/finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"459bb9214094ea8f"}},{"code_sha256_prefix":"0713ce4249b75fac","entry":"process_batch","repo":"THUDM/CogAgent","repo_kind":"official","path":"finetune/finetune.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/finetune/finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0713ce4249b75fac"}},{"code_sha256_prefix":"0c9a09bd7e2934b7","entry":"process_batch_eval","repo":"THUDM/CogAgent","repo_kind":"official","path":"finetune/finetune.py","file_url":"https://github.com/THUDM/CogAgent/blob/HEAD/finetune/finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c9a09bd7e2934b7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}