{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/webrl-training-llm-web-agents-via-self","title":"WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning","arxiv_id":"2411.02337","date":"2024-11-04","proceeding":null,"authors":["Zehan Qi","Xiao Liu","Iat Long Iong","Hanyu Lai","Xueqiao Sun","Wenyi Zhao","Yu Yang","Xinyue Yang","Jiadai Sun","Shuntian Yao","Tianjie Zhang","Wei Xu","Jie Tang","Yuxiao Dong"],"abstract":"Large language models (LLMs) have shown remarkable potential as autonomous agents, particularly in web-based tasks. However, existing LLM web agents heavily rely on expensive proprietary LLM APIs, while open LLMs lack the necessary decision-making capabilities. This paper introduces WebRL, a self-evolving online curriculum reinforcement learning framework designed to train high-performance web agents using open LLMs. WebRL addresses three key challenges in building LLM web agents, including the scarcity of training tasks, sparse feedback signals, and policy distribution drift in online learning. Specifically, WebRL incorporates 1) a self-evolving curriculum that generates new tasks from unsuccessful attempts, 2) a robust outcome-supervised reward model (ORM), and 3) adaptive reinforcement learning strategies to ensure consistent improvements. We apply WebRL to transform open Llama-3.1 and GLM-4 models into proficient web agents. On WebArena-Lite, WebRL improves the success rate of Llama-3.1-8B from 4.8% to 42.4%, and from 6.1% to 43% for GLM-4-9B. These open models significantly surpass the performance of GPT-4-Turbo (17.6%) and GPT-4o (13.9%) and outperform previous state-of-the-art web agents trained on open LLMs (AutoWebGLM, 18.2%). Our findings demonstrate WebRL's effectiveness in bridging the gap between open and proprietary LLM-based web agents, paving the way for more accessible and powerful autonomous web interaction systems.","url_abs":"https://arxiv.org/abs/2411.02337v3","url_pdf":"https://arxiv.org/pdf/2411.02337v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"webrl-training-llm-web-agents-via-self","repo_url":"https://github.com/THUDM/WebRL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.02337","atlas_url":"https://app.syntology.ai/?focus=2411.02337","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.02337"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/THUDM/WebRL","reach":{"status":"ok"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"6bc27263e082aea0","entry":"add_mc_return","repo":"THUDM/WebRL","repo_kind":"official","path":"webrl/environment/env_utils.py","file_url":"https://github.com/THUDM/WebRL/blob/HEAD/webrl/environment/env_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6bc27263e082aea0"}},{"code_sha256_prefix":"31a5caa2d17b76d5","entry":"add_trajectory_reward","repo":"THUDM/WebRL","repo_kind":"official","path":"webrl/environment/env_utils.py","file_url":"https://github.com/THUDM/WebRL/blob/HEAD/webrl/environment/env_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31a5caa2d17b76d5"}},{"code_sha256_prefix":"32579324c0d766ef","entry":"dict_mean","repo":"THUDM/WebRL","repo_kind":"official","path":"webrl/algorithms/webrl/trainer.py","file_url":"https://github.com/THUDM/WebRL/blob/HEAD/webrl/algorithms/webrl/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32579324c0d766ef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}