{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/universal-adversarial-perturbations-for-3","title":"Universal Adversarial Perturbations for Vision-Language Pre-trained Models","arxiv_id":"2405.05524","date":"2024-05-09","proceeding":null,"authors":["Peng-Fei Zhang","Zi Huang","Guangdong Bai"],"abstract":"Vision-language pre-trained (VLP) models have been the foundation of numerous vision-language tasks. Given their prevalence, it be- comes imperative to assess their adversarial robustness, especially when deploying them in security-crucial real-world applications. Traditionally, adversarial perturbations generated for this assessment target specific VLP models, datasets, and/or downstream tasks. This practice suffers from low transferability and additional computation costs when transitioning to new scenarios. In this work, we thoroughly investigate whether VLP models are commonly sensitive to imperceptible perturbations of a specific pattern for the image modality. To this end, we propose a novel black-box method to generate Universal Adversarial Perturbations (UAPs), which is so called the Effective and T ransferable Universal Adversarial Attack (ETU), aiming to mislead a variety of existing VLP models in a range of downstream tasks. The ETU comprehensively takes into account the characteristics of UAPs and the intrinsic cross-modal interactions to generate effective UAPs. Under this regime, the ETU encourages both global and local utilities of UAPs. This benefits the overall utility while reducing interactions between UAP units, improving the transferability. To further enhance the effectiveness and transferability of UAPs, we also design a novel data augmentation method named ScMix. ScMix consists of self-mix and cross-mix data transformations, which can effectively increase the multi-modal data diversity while preserving the semantics of the original data. Through comprehensive experiments on various downstream tasks, VLP models, and datasets, we demonstrate that the proposed method is able to achieve effective and transferrable universal adversarial attacks.","url_abs":"https://arxiv.org/abs/2405.05524v1","url_pdf":"https://arxiv.org/pdf/2405.05524v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"universal-adversarial-perturbations-for-3","repo_url":"https://github.com/HanxunH/XTransferBench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"universal-adversarial-perturbations-for-3","repo_url":"https://github.com/sduzpf/UAP_VLP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-attack","task_name":"Adversarial Attack"},{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.05524","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.05524"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sduzpf/UAP_VLP","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HanxunH/XTransferBench","reach":null}],"summary":{"ran_violates":1,"unverified":1},"by_repo_kind":{"listed":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"734f672cf749108b","entry":"input_diversity","repo":"sduzpf/UAP_VLP","repo_kind":"listed","path":"Eval_Retrieval_CLIP.py","file_url":"https://github.com/sduzpf/UAP_VLP/blob/HEAD/Eval_Retrieval_CLIP.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"734f672cf749108b"}},{"code_sha256_prefix":"7b5d3ea7c09e78d8","entry":"random_init","repo":"sduzpf/UAP_VLP","repo_kind":"listed","path":"Eval_Retrieval_CLIP.py","file_url":"https://github.com/sduzpf/UAP_VLP/blob/HEAD/Eval_Retrieval_CLIP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b5d3ea7c09e78d8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}