{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reinforcement-learning-on-web-interfaces","title":"Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration","arxiv_id":"1802.08802","date":"2018-02-24","proceeding":"ICLR 2018 1","authors":["Evan Zheran Liu","Kelvin Guu","Panupong Pasupat","Tianlin Shi","Percy Liang"],"abstract":"Reinforcement learning (RL) agents improve through trial-and-error, but when\nreward is sparse and the agent cannot discover successful action sequences,\nlearning stagnates. This has been a notable problem in training deep RL agents\nto perform web-based tasks, such as booking flights or replying to emails,\nwhere a single mistake can ruin the entire sequence of actions. A common remedy\nis to \"warm-start\" the agent by pre-training it to mimic expert demonstrations,\nbut this is prone to overfitting. Instead, we propose to constrain exploration\nusing demonstrations. From each demonstration, we induce high-level \"workflows\"\nwhich constrain the allowable actions at each time step to be similar to those\nin the demonstration (e.g., \"Step 1: click on a textbox; Step 2: enter some\ntext\"). Our exploration policy then learns to identify successful workflows and\nsamples actions that satisfy these workflows. Workflows prune out bad\nexploration directions and accelerate the agent's ability to discover rewards.\nWe use our approach to train a novel neural policy designed to handle the\nsemi-structured nature of websites, and evaluate on a suite of web tasks,\nincluding the recent World of Bits benchmark. We achieve new state-of-the-art\nresults, and show that workflow-guided exploration improves sample efficiency\nover behavioral cloning by more than 100x.","url_abs":"http://arxiv.org/abs/1802.08802v1","url_pdf":"http://arxiv.org/pdf/1802.08802v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reinforcement-learning-on-web-interfaces","repo_url":"https://github.com/stanfordnlp/wge","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"reinforcement-learning-on-web-interfaces","repo_url":"https://worksheets.codalab.org/worksheets/0x0f25031bd42f4aabbc17625fe1484066","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"reinforcement-learning-on-web-interfaces","repo_url":"https://github.com/farama-foundation/miniwob-plusplus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"reinforcement-learning-on-web-interfaces","repo_url":"https://github.com/google-research/android_world","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"reinforcement-learning-on-web-interfaces","repo_url":"https://github.com/zbyte64/pytorch-fuzzdom","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1802.08802","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1802.08802"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stanfordnlp/wge","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/farama-foundation/miniwob-plusplus","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zbyte64/pytorch-fuzzdom","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://worksheets.codalab.org/worksheets/0x0f25031bd42f4aabbc17625fe1484066","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/android_world","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":9},"by_repo_kind":{"listed":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"13fb0b9d4d6f6cbd","entry":"create_empty_screenshot","repo":"farama-foundation/miniwob-plusplus","repo_kind":"listed","path":"miniwob/observation.py","file_url":"https://github.com/farama-foundation/miniwob-plusplus/blob/HEAD/miniwob/observation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"13fb0b9d4d6f6cbd"}},{"code_sha256_prefix":"98e7598c3b6a22d3","entry":"create_regex_field_extractor","repo":"farama-foundation/miniwob-plusplus","repo_kind":"listed","path":"miniwob/fields.py","file_url":"https://github.com/farama-foundation/miniwob-plusplus/blob/HEAD/miniwob/fields.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"98e7598c3b6a22d3"}},{"code_sha256_prefix":"3b64b39bfe55372b","entry":"field_lookup","repo":"farama-foundation/miniwob-plusplus","repo_kind":"listed","path":"miniwob/fields.py","file_url":"https://github.com/farama-foundation/miniwob-plusplus/blob/HEAD/miniwob/fields.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"3b64b39bfe55372b"}},{"code_sha256_prefix":"8ab9d5590a1d2e88","entry":"get_binary_reward","repo":"farama-foundation/miniwob-plusplus","repo_kind":"listed","path":"miniwob/reward.py","file_url":"https://github.com/farama-foundation/miniwob-plusplus/blob/HEAD/miniwob/reward.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"8ab9d5590a1d2e88"}},{"code_sha256_prefix":"75da28078a8a385c","entry":"get_field_extractor","repo":"farama-foundation/miniwob-plusplus","repo_kind":"listed","path":"miniwob/fields.py","file_url":"https://github.com/farama-foundation/miniwob-plusplus/blob/HEAD/miniwob/fields.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"75da28078a8a385c"}},{"code_sha256_prefix":"5138670e97d11062","entry":"get_original_reward","repo":"farama-foundation/miniwob-plusplus","repo_kind":"listed","path":"miniwob/reward.py","file_url":"https://github.com/farama-foundation/miniwob-plusplus/blob/HEAD/miniwob/reward.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"5138670e97d11062"}},{"code_sha256_prefix":"ad7d838b3aca1f14","entry":"get_raw_reward","repo":"farama-foundation/miniwob-plusplus","repo_kind":"listed","path":"miniwob/reward.py","file_url":"https://github.com/farama-foundation/miniwob-plusplus/blob/HEAD/miniwob/reward.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"ad7d838b3aca1f14"}},{"code_sha256_prefix":"f561b96d1f313655","entry":"pil_to_numpy_array","repo":"farama-foundation/miniwob-plusplus","repo_kind":"listed","path":"miniwob/screenshot.py","file_url":"https://github.com/farama-foundation/miniwob-plusplus/blob/HEAD/miniwob/screenshot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"f561b96d1f313655"}},{"code_sha256_prefix":"5bcb6817b5a7ba8b","entry":"start_http_server","repo":"farama-foundation/miniwob-plusplus","repo_kind":"listed","path":"miniwob/http_server.py","file_url":"https://github.com/farama-foundation/miniwob-plusplus/blob/HEAD/miniwob/http_server.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"5bcb6817b5a7ba8b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}