{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vem-environment-free-exploration-for-training","title":"VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model","arxiv_id":"2502.18906","date":"2025-02-26","proceeding":null,"authors":["Jiani Zheng","Lu Wang","Fangkai Yang","Chaoyun Zhang","Lingrui Mei","Wenjie Yin","QIngwei Lin","Dongmei Zhang","Saravan Rajmohan","Qi Zhang"],"abstract":"Training Vision-Language Models (VLMs) for Graphical User Interfaces (GUI) agents via Reinforcement Learning (RL) faces critical challenges: environment-based RL requires costly interactions, while environment-free methods struggle with distribution shift and reward generalization. We propose an environment-free RL framework that decouples value estimation from policy optimization by leveraging a pretrained Value Environment Model (VEM). VEM predicts state-action values directly from offline data, distilling human-like priors about GUI interaction outcomes without requiring next-state prediction or environmental feedback. This avoids compounding errors and enhances resilience to UI changes by focusing on semantic reasoning (e.g., Does this action advance the user's goal?). The framework operates in two stages: (1) pretraining VEM to estimate long-term action utilities and (2) guiding policy exploration with frozen VEM signals, enabling layout-agnostic GUI automation. Evaluated on Android-in-the-Wild benchmarks, VEM achieves state-of-the-art performance in both offline and online settings, outperforming environment-free baselines significantly and matching environment-based approaches without interaction costs. Importantly, VEM demonstrates that semantic-aware value estimation can achieve comparable performance with online-trained methods.","url_abs":"https://arxiv.org/abs/2502.18906v1","url_pdf":"https://arxiv.org/pdf/2502.18906v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vem-environment-free-exploration-for-training","repo_url":"https://github.com/microsoft/gui-agent-rl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.18906","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.18906"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/gui-agent-rl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":10},"by_repo_kind":{"listed":{"samples":11,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"41f6e1b0feaded2f","entry":"is_tap_action","repo":"microsoft/gui-agent-rl","repo_kind":"listed","path":"eval_tools/metrix.py","file_url":"https://github.com/microsoft/gui-agent-rl/blob/HEAD/eval_tools/metrix.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41f6e1b0feaded2f"}},{"code_sha256_prefix":"bfea191ff4c26791","entry":"action_dict_to_class","repo":"microsoft/gui-agent-rl","repo_kind":"listed","path":"data_preprocess/utils.py","file_url":"https://github.com/microsoft/gui-agent-rl/blob/HEAD/data_preprocess/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bfea191ff4c26791"}},{"code_sha256_prefix":"d5a7ab5e7ead7d94","entry":"build_prompt_general","repo":"microsoft/gui-agent-rl","repo_kind":"listed","path":"data_preprocess/prompt.py","file_url":"https://github.com/microsoft/gui-agent-rl/blob/HEAD/data_preprocess/prompt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d5a7ab5e7ead7d94"}},{"code_sha256_prefix":"6021d081bc8737c4","entry":"check_actions_match","repo":"microsoft/gui-agent-rl","repo_kind":"listed","path":"eval_tools/metrix.py","file_url":"https://github.com/microsoft/gui-agent-rl/blob/HEAD/eval_tools/metrix.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6021d081bc8737c4"}},{"code_sha256_prefix":"7802770e6818ba35","entry":"encode_image","repo":"microsoft/gui-agent-rl","repo_kind":"listed","path":"data_preprocess/gpt.py","file_url":"https://github.com/microsoft/gui-agent-rl/blob/HEAD/data_preprocess/gpt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7802770e6818ba35"}},{"code_sha256_prefix":"4057e0b6782bb278","entry":"extract_scroll","repo":"microsoft/gui-agent-rl","repo_kind":"listed","path":"data_preprocess/utils.py","file_url":"https://github.com/microsoft/gui-agent-rl/blob/HEAD/data_preprocess/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4057e0b6782bb278"}},{"code_sha256_prefix":"d6133035efb0d990","entry":"get_gpt_4o","repo":"microsoft/gui-agent-rl","repo_kind":"listed","path":"data_preprocess/gpt.py","file_url":"https://github.com/microsoft/gui-agent-rl/blob/HEAD/data_preprocess/gpt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6133035efb0d990"}},{"code_sha256_prefix":"9c771dc2b4ad47bd","entry":"get_message","repo":"microsoft/gui-agent-rl","repo_kind":"listed","path":"data_preprocess/gpt.py","file_url":"https://github.com/microsoft/gui-agent-rl/blob/HEAD/data_preprocess/gpt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c771dc2b4ad47bd"}},{"code_sha256_prefix":"ff74d412d9163765","entry":"parse_response","repo":"microsoft/gui-agent-rl","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/microsoft/gui-agent-rl/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff74d412d9163765"}},{"code_sha256_prefix":"bf2b970192f06b31","entry":"read_json","repo":"microsoft/gui-agent-rl","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/microsoft/gui-agent-rl/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf2b970192f06b31"}},{"code_sha256_prefix":"1fea86da69ed788a","entry":"str_2_format","repo":"microsoft/gui-agent-rl","repo_kind":"listed","path":"eval_tools/metrix.py","file_url":"https://github.com/microsoft/gui-agent-rl/blob/HEAD/eval_tools/metrix.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1fea86da69ed788a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}