{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/screenagent-a-vision-language-model-driven","title":"ScreenAgent: A Vision Language Model-driven Computer Control Agent","arxiv_id":"2402.07945","date":"2024-02-09","proceeding":null,"authors":["Runliang Niu","Jindong Li","Shiqi Wang","Yali Fu","Xiyu Hu","Xueyuan Leng","He Kong","Yi Chang","Qi Wang"],"abstract":"Existing Large Language Models (LLM) can invoke a variety of tools and APIs to complete complex tasks. The computer, as the most powerful and universal tool, could potentially be controlled directly by a trained LLM agent. Powered by the computer, we can hopefully build a more generalized agent to assist humans in various daily digital works. In this paper, we construct an environment for a Vision Language Model (VLM) agent to interact with a real computer screen. Within this environment, the agent can observe screenshots and manipulate the Graphics User Interface (GUI) by outputting mouse and keyboard actions. We also design an automated control pipeline that includes planning, acting, and reflecting phases, guiding the agent to continuously interact with the environment and complete multi-step tasks. Additionally, we construct the ScreenAgent Dataset, which collects screenshots and action sequences when completing a variety of daily computer tasks. Finally, we trained a model, ScreenAgent, which achieved computer control capabilities comparable to GPT-4V and demonstrated more precise UI positioning capabilities. Our attempts could inspire further research on building a generalist LLM agent. The code is available at \\url{https://github.com/niuzaisheng/ScreenAgent}.","url_abs":"https://arxiv.org/abs/2402.07945v1","url_pdf":"https://arxiv.org/pdf/2402.07945v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"screenagent-a-vision-language-model-driven","repo_url":"https://github.com/niuzaisheng/screenagent","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.07945","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.07945"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/niuzaisheng/screenagent","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/niuzaisheng/ScreenAgent","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":6,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"49289a7f94c227bd","entry":"compute_bleu","repo":"niuzaisheng/ScreenAgent","repo_kind":"official","path":"client/bleu.py","file_url":"https://github.com/niuzaisheng/ScreenAgent/blob/HEAD/client/bleu.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"49289a7f94c227bd"}},{"code_sha256_prefix":"ad5ba95300a1cc7a","entry":"data_collator","repo":"niuzaisheng/ScreenAgent","repo_kind":"official","path":"train/finetune_ScreenAgent.py","file_url":"https://github.com/niuzaisheng/ScreenAgent/blob/HEAD/train/finetune_ScreenAgent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ad5ba95300a1cc7a"}},{"code_sha256_prefix":"6493522dff8f8fa5","entry":"load_image_from_base64","repo":"niuzaisheng/ScreenAgent","repo_kind":"official","path":"train/cogagent_model_worker.py","file_url":"https://github.com/niuzaisheng/ScreenAgent/blob/HEAD/train/cogagent_model_worker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6493522dff8f8fa5"}},{"code_sha256_prefix":"625f66bc3b7b4b5b","entry":"pack_ard","repo":"niuzaisheng/ScreenAgent","repo_kind":"official","path":"client/asyncvnc.py","file_url":"https://github.com/niuzaisheng/ScreenAgent/blob/HEAD/client/asyncvnc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"625f66bc3b7b4b5b"}},{"code_sha256_prefix":"6741565cee168112","entry":"read_int","repo":"niuzaisheng/ScreenAgent","repo_kind":"official","path":"client/asyncvnc.py","file_url":"https://github.com/niuzaisheng/ScreenAgent/blob/HEAD/client/asyncvnc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6741565cee168112"}},{"code_sha256_prefix":"52dc9a4327ee52bd","entry":"read_text","repo":"niuzaisheng/ScreenAgent","repo_kind":"official","path":"client/asyncvnc.py","file_url":"https://github.com/niuzaisheng/ScreenAgent/blob/HEAD/client/asyncvnc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"52dc9a4327ee52bd"}},{"code_sha256_prefix":"c87254327d5ba881","entry":"convert_mouse_button","repo":"niuzaisheng/ScreenAgent","repo_kind":"official","path":"client/base.py","file_url":"https://github.com/niuzaisheng/ScreenAgent/blob/HEAD/client/base.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c87254327d5ba881"}},{"code_sha256_prefix":"3679d1592673c770","entry":"find_non_json_span_from_text","repo":"niuzaisheng/ScreenAgent","repo_kind":"official","path":"client/action.py","file_url":"https://github.com/niuzaisheng/ScreenAgent/blob/HEAD/client/action.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3679d1592673c770"}},{"code_sha256_prefix":"6bf13d2e007c2458","entry":"split_json_span_and_non_json_span_from_text","repo":"niuzaisheng/ScreenAgent","repo_kind":"official","path":"client/action.py","file_url":"https://github.com/niuzaisheng/ScreenAgent/blob/HEAD/client/action.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6bf13d2e007c2458"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}