{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-06367","title":"WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks","arxiv_id":"2604.06367","date":"2026-04-07","proceeding":null,"authors":["Guruprasad Viswanathan Ramesh","Asmit Nayak","Basieem Siddique","Kassem Fawaz"],"abstract":"Web agents automate browser tasks, ranging from simple form completion to complex workflows like ordering groceries. While current benchmarks evaluate general-purpose performance~(e.g., WebArena) or safety against malicious actions~(e.g., SafeArena), no existing framework assesses an agent's ability to successfully execute user-facing website security and privacy tasks, such as managing cookie preferences, configuring privacy-sensitive account settings, or revoking inactive sessions. To address this gap, we introduce WebSP-Eval, an evaluation framework for measuring web agent performance on website security and privacy tasks. WebSP-Eval comprises 1) a manually crafted task dataset of 200 task instances across 28 websites; 2) a robust agentic system supporting account and initial state management across runs using a custom Google Chrome extension; and 3) an automated evaluator. We evaluate a total of 8 web agent instantiations using state-of-the-art multimodal large language models, conducting a fine-grained analysis across websites, task categories, and UI elements. Our evaluation reveals that current models suffer from limited autonomous exploration capabilities to reliably solve website security and privacy tasks, and struggle with specific task categories and websites. Crucially, we identify stateful UI elements are a primary reason for agent failure, with toggles causing more than 45% task failure across many models.","url_abs":"https://arxiv.org/abs/2604.06367","url_pdf":"https://arxiv.org/pdf/2604.06367","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.06367","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.06367"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/wi-pi/webspeval_code","reach":null}],"summary":{"unverified":15},"by_repo_kind":{"found_in_text":{"samples":15,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8eca7eb89de5f8fe","entry":"auto_eval_by_claude","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"judge/eval_binary_with_gt_text_claude.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/judge/eval_binary_with_gt_text_claude.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8eca7eb89de5f8fe"}},{"code_sha256_prefix":"c8622f34c1208025","entry":"auto_eval_by_gpt5","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"judge/eval_binary_with_gt_text_gpt5.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/judge/eval_binary_with_gt_text_gpt5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c8622f34c1208025"}},{"code_sha256_prefix":"2881b7afbbf74c18","entry":"check_for_api_errors","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"judge/eval_binary_with_gt_text_claude.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/judge/eval_binary_with_gt_text_claude.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2881b7afbbf74c18"}},{"code_sha256_prefix":"5205d09c9c182848","entry":"create_fresh_profile","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"src/agent/driver.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/src/agent/driver.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5205d09c9c182848"}},{"code_sha256_prefix":"66d017c4b55d2321","entry":"encode_image_base64","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"judge/eval_binary_with_gt_text_claude.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/judge/eval_binary_with_gt_text_claude.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66d017c4b55d2321"}},{"code_sha256_prefix":"282e8a7fe50932e3","entry":"fetch_browser_info","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"src/utils_webarena.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/src/utils_webarena.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"282e8a7fe50932e3"}},{"code_sha256_prefix":"a5524a41e66f9df4","entry":"format_msg_claude","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"src/api_utils.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/src/api_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a5524a41e66f9df4"}},{"code_sha256_prefix":"76510fc830c652c8","entry":"format_msg_gpt5","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"src/api_utils.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/src/api_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"76510fc830c652c8"}},{"code_sha256_prefix":"cf52e4eb5297bbbc","entry":"format_msg_text_only_gpt5","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"src/api_utils.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/src/api_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf52e4eb5297bbbc"}},{"code_sha256_prefix":"040b32129bf67f60","entry":"get_bounding_client_rect","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"src/utils_webarena.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/src/utils_webarena.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"040b32129bf67f60"}},{"code_sha256_prefix":"0b27cebef7f8500d","entry":"get_element_in_viewport_ratio","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"src/utils_webarena.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/src/utils_webarena.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b27cebef7f8500d"}},{"code_sha256_prefix":"602e1a57b115b9ef","entry":"get_tabs_info","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"src/agent/actions.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/src/agent/actions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"602e1a57b115b9ef"}},{"code_sha256_prefix":"53bdbb7cef5156de","entry":"load_json_file","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"src/state_reset_check.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/src/state_reset_check.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"53bdbb7cef5156de"}},{"code_sha256_prefix":"e63c4f0834fe9c79","entry":"resize_image","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"src/utils.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e63c4f0834fe9c79"}},{"code_sha256_prefix":"c0d2dc95bd1e83ea","entry":"switch_to_latest_tab","repo":"wi-pi/webspeval_code","repo_kind":"found_in_text","path":"src/agent/actions.py","file_url":"https://github.com/wi-pi/webspeval_code/blob/HEAD/src/agent/actions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c0d2dc95bd1e83ea"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}