{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vlattack-multimodal-adversarial-attacks-on","title":"VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models","arxiv_id":"2310.04655","date":"2023-10-07","proceeding":"NeurIPS 2023 11","authors":["Ziyi Yin","Muchao Ye","Tianrong Zhang","Tianyu Du","Jinguo Zhu","Han Liu","Jinghui Chen","Ting Wang","Fenglong Ma"],"abstract":"Vision-Language (VL) pre-trained models have shown their superiority on many multimodal tasks. However, the adversarial robustness of such models has not been fully explored. Existing approaches mainly focus on exploring the adversarial robustness under the white-box setting, which is unrealistic. In this paper, we aim to investigate a new yet practical task to craft image and text perturbations using pre-trained VL models to attack black-box fine-tuned models on different downstream tasks. Towards this end, we propose VLATTACK to generate adversarial samples by fusing perturbations of images and texts from both single-modal and multimodal levels. At the single-modal level, we propose a new block-wise similarity attack (BSA) strategy to learn image perturbations for disrupting universal representations. Besides, we adopt an existing text attack strategy to generate text perturbations independent of the image-modal attack. At the multimodal level, we design a novel iterative cross-search attack (ICSA) method to update adversarial image-text pairs periodically, starting with the outputs from the single-modal level. We conduct extensive experiments to attack five widely-used VL pre-trained models for six tasks. Experimental results show that VLATTACK achieves the highest attack success rates on all tasks compared with state-of-the-art baselines, which reveals a blind spot in the deployment of pre-trained VL models. Source codes can be found at https://github.com/ericyinyzy/VLAttack.","url_abs":"https://arxiv.org/abs/2310.04655v3","url_pdf":"https://arxiv.org/pdf/2310.04655v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vlattack-multimodal-adversarial-attacks-on","repo_url":"https://github.com/ericyinyzy/vlattack","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.04655","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.04655"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ericyinyzy/vlattack","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"aa5a91226269bf75","entry":"tile","repo":"ericyinyzy/vlattack","repo_kind":"official","path":"BLIP_attack/models/blip_vqa.py","file_url":"https://github.com/ericyinyzy/vlattack/blob/HEAD/BLIP_attack/models/blip_vqa.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5a91226269bf75"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"ericyinyzy/vlattack","repo_kind":"official","path":"BLIP_attack/models/blip_retrieval.py","file_url":"https://github.com/ericyinyzy/vlattack/blob/HEAD/BLIP_attack/models/blip_retrieval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"ericyinyzy/vlattack","repo_kind":"official","path":"BLIP_attack/models/blip_pretrain.py","file_url":"https://github.com/ericyinyzy/vlattack/blob/HEAD/BLIP_attack/models/blip_pretrain.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"8bad2bfb151fd2a8","entry":"find_clip_right","repo":"ericyinyzy/VLAttack","repo_kind":"official","path":"CLIP_attack/clip_attack.py","file_url":"https://github.com/ericyinyzy/VLAttack/blob/HEAD/CLIP_attack/clip_attack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8bad2bfb151fd2a8"}},{"code_sha256_prefix":"5691c1c466b2b0ff","entry":"pgd_attack","repo":"ericyinyzy/VLAttack","repo_kind":"official","path":"CLIP_attack/clip_attack.py","file_url":"https://github.com/ericyinyzy/VLAttack/blob/HEAD/CLIP_attack/clip_attack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5691c1c466b2b0ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}