{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/st-webagentbench-a-benchmark-for-evaluating","title":"ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents","arxiv_id":"2410.06703","date":"2024-10-09","proceeding":null,"authors":["Ido Levy","Ben wiesel","Sami Marreed","Alon Oved","Avi Yaeli","Segev Shlomov"],"abstract":"Autonomous web agents solve complex browsing tasks, yet existing benchmarks measure only whether an agent finishes a task, ignoring whether it does so safely or in a way enterprises can trust. To integrate these agents into critical workflows, safety and trustworthiness (ST) are prerequisite conditions for adoption. We introduce \\textbf{\\textsc{ST-WebAgentBench}}, a configurable and easily extensible suite for evaluating web agent ST across realistic enterprise scenarios. Each of its 222 tasks is paired with ST policies, concise rules that encode constraints, and is scored along six orthogonal dimensions (e.g., user consent, robustness). Beyond raw task success, we propose the \\textit{Completion Under Policy} (\\textit{CuP}) metric, which credits only completions that respect all applicable policies, and the \\textit{Risk Ratio}, which quantifies ST breaches across dimensions. Evaluating three open state-of-the-art agents reveals that their average CuP is less than two-thirds of their nominal completion rate, exposing critical safety gaps. By releasing code, evaluation templates, and a policy-authoring interface, \\href{https://sites.google.com/view/st-webagentbench/home}{\\textsc{ST-WebAgentBench}} provides an actionable first step toward deploying trustworthy web agents at scale.","url_abs":"https://arxiv.org/abs/2410.06703v4","url_pdf":"https://arxiv.org/pdf/2410.06703v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"st-webagentbench-a-benchmark-for-evaluating","repo_url":"https://github.com/segev-shlomov/ST-WebAgentBench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"autonomous-web-navigation","task_name":"Autonomous Web Navigation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.06703","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.06703"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/segev-shlomov/ST-WebAgentBench","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"a42579504bbb0797","entry":"compute_completion_rates","repo":"segev-shlomov/ST-WebAgentBench","repo_kind":"listed","path":"stwebagentbench/result_analysis/analyze.py","file_url":"https://github.com/segev-shlomov/ST-WebAgentBench/blob/HEAD/stwebagentbench/result_analysis/analyze.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a42579504bbb0797"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}