{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-08005","title":"Preference Redirection via Attention Concentration: An Attack on Computer Use Agents","arxiv_id":"2604.08005","date":"2026-04-09","proceeding":null,"authors":["Dominik Seip","Matthias Hein"],"abstract":"Advancements in multimodal foundation models have enabled the development of Computer Use Agents (CUAs) capable of autonomously interacting with GUI environments. As CUAs are not restricted to certain tools, they allow to automate more complex agentic tasks but at the same time open up new security vulnerabilities. While prior work has concentrated on the language modality, the vulnerability of the vision modality has received less attention. In this paper, we introduce PRAC, a novel attack that, unlike prior work targeting the VLM output directly, manipulates the model's internal preferences by redirecting its attention toward a stealthy adversarial patch. We show that PRAC is able to manipulate the selection process of a CUA on an online shopping platform towards a chosen target product. While we require white-box access to the model for the creation of the attack, we show that our attack generalizes to fine-tuned versions of the same model, presenting a critical threat as multiple companies build specific CUAs based on open weights models.","url_abs":"https://arxiv.org/abs/2604.08005","url_pdf":"https://arxiv.org/pdf/2604.08005","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.08005","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.08005"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Dominik5431/PRAC","reach":null}],"summary":{"unverified":18},"by_repo_kind":{"found_in_text":{"samples":18,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"97496330a30f8c79","entry":"apply_mask_to_padding_states","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/attack/modeling_qwen3_5.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/attack/modeling_qwen3_5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"97496330a30f8c79"}},{"code_sha256_prefix":"f9a0c5fae6908b08","entry":"build_conversation_prompt","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/models/internvl_differentiable_processor.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/models/internvl_differentiable_processor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f9a0c5fae6908b08"}},{"code_sha256_prefix":"24570f95db64ff2a","entry":"compute_attention_distributions","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/evaluate_attention.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/evaluate_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"24570f95db64ff2a"}},{"code_sha256_prefix":"f09c9a6ad6b42c01","entry":"compute_attention_score","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/evaluate_attention.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/evaluate_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f09c9a6ad6b42c01"}},{"code_sha256_prefix":"25e97d41b0bd28a8","entry":"gaussian_defense","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/evaluate_defenses.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/evaluate_defenses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"25e97d41b0bd28a8"}},{"code_sha256_prefix":"ed6f9f96e3bd7c19","entry":"get_image_size","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/models/glm_differentiable_processor.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/models/glm_differentiable_processor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ed6f9f96e3bd7c19"}},{"code_sha256_prefix":"8f56b3172e16fbda","entry":"get_internvl_conv_template","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/models/internvl_differentiable_processor.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/models/internvl_differentiable_processor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8f56b3172e16fbda"}},{"code_sha256_prefix":"ea4b01bb60b39a62","entry":"get_prompt_embedding","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/evaluate_transfer_prompt.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/evaluate_transfer_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ea4b01bb60b39a62"}},{"code_sha256_prefix":"07351df0c4ca0a61","entry":"image_to_base64","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/evaluate.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"07351df0c4ca0a61"}},{"code_sha256_prefix":"4284e36daa3a10f1","entry":"is_gemini_model","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/models/inference_wrapper.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/models/inference_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4284e36daa3a10f1"}},{"code_sha256_prefix":"24900138e367d941","entry":"is_openai_model","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/models/inference_wrapper.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/models/inference_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"24900138e367d941"}},{"code_sha256_prefix":"ad02a4d73e288c06","entry":"l2norm","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/attack/modeling_qwen3_5.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/attack/modeling_qwen3_5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ad02a4d73e288c06"}},{"code_sha256_prefix":"f2313e5b3e28311a","entry":"make_batched_images","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/models/internvl_differentiable_processor.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/models/internvl_differentiable_processor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f2313e5b3e28311a"}},{"code_sha256_prefix":"a9cd97bd7891399b","entry":"rescale","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/models/glm_differentiable_processor.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/models/glm_differentiable_processor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a9cd97bd7891399b"}},{"code_sha256_prefix":"1c335eea3b5ed973","entry":"resize","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/models/glm_differentiable_processor.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/models/glm_differentiable_processor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1c335eea3b5ed973"}},{"code_sha256_prefix":"908373b4f3e8c6d5","entry":"sample_prompts_at_distance","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/evaluate_transfer_prompt.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/evaluate_transfer_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"908373b4f3e8c6d5"}},{"code_sha256_prefix":"92a19e868b324387","entry":"strip_thinking","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/models/inference_wrapper.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/models/inference_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"92a19e868b324387"}},{"code_sha256_prefix":"f7518069bfc15c10","entry":"torch_causal_conv1d_update","repo":"Dominik5431/PRAC","repo_kind":"found_in_text","path":"prac/attack/modeling_qwen3_5.py","file_url":"https://github.com/Dominik5431/PRAC/blob/HEAD/prac/attack/modeling_qwen3_5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f7518069bfc15c10"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}