{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/one-perturbation-is-enough-on-generating","title":"One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models","arxiv_id":"2406.05491","date":"2024-06-08","proceeding":null,"authors":["Hao Fang","Jiawei Kong","Wenbo Yu","Bin Chen","Jiawei Li","Hao Wu","Shutao Xia","Ke Xu"],"abstract":"Vision-Language Pre-training (VLP) models have exhibited unprecedented capability in many applications by taking full advantage of the multimodal alignment. However, previous studies have shown they are vulnerable to maliciously crafted adversarial samples. Despite recent success, these methods are generally instance-specific and require generating perturbations for each input sample. In this paper, we reveal that VLP models are also vulnerable to the instance-agnostic universal adversarial perturbation (UAP). Specifically, we design a novel Contrastive-training Perturbation Generator with Cross-modal conditions (C-PGC) to achieve the attack. In light that the pivotal multimodal alignment is achieved through the advanced contrastive learning technique, we devise to turn this powerful weapon against themselves, i.e., employ a malicious version of contrastive learning to train the C-PGC based on our carefully crafted positive and negative image-text pairs for essentially destroying the alignment relationship learned by VLP models. Besides, C-PGC fully utilizes the characteristics of Vision-and-Language (V+L) scenarios by incorporating both unimodal and cross-modal information as effective guidance. Extensive experiments show that C-PGC successfully forces adversarial samples to move away from their original area in the VLP model's feature space, thus essentially enhancing attacks across various victim models and V+L tasks. The GitHub repository is available at https://github.com/ffhibnese/CPGC_VLP_Universal_Attacks.","url_abs":"https://arxiv.org/abs/2406.05491v3","url_pdf":"https://arxiv.org/pdf/2406.05491v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"one-perturbation-is-enough-on-generating","repo_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"one-perturbation-is-enough-on-generating","repo_url":"https://github.com/HanxunH/XTransferBench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.05491","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.05491"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HanxunH/XTransferBench","reach":null}],"summary":{"ran_draft_wrong":5,"ran_fixture":1,"ran_violates":2,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"cf9ffa02a42184af","entry":"whitespace_tokenize","repo":"ffhibnese/cpgc_vlp_universal_attacks","repo_kind":"official","path":"models/tokenization_bert.py","file_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks/blob/HEAD/models/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cf9ffa02a42184af"}},{"code_sha256_prefix":"aa81d03f07735ab8","entry":"compute_acc","repo":"ffhibnese/cpgc_vlp_universal_attacks","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa81d03f07735ab8"}},{"code_sha256_prefix":"ad12a494674d23fb","entry":"compute_n_params","repo":"ffhibnese/cpgc_vlp_universal_attacks","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad12a494674d23fb"}},{"code_sha256_prefix":"424012cb37b31172","entry":"default","repo":"ffhibnese/cpgc_vlp_universal_attacks","repo_kind":"official","path":"attention.py","file_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks/blob/HEAD/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"424012cb37b31172"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"ffhibnese/cpgc_vlp_universal_attacks","repo_kind":"official","path":"attention.py","file_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks/blob/HEAD/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"c6ec173f19f5c34d","entry":"interpolate_pos_embed","repo":"ffhibnese/cpgc_vlp_universal_attacks","repo_kind":"official","path":"models/vit.py","file_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks/blob/HEAD/models/vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6ec173f19f5c34d"}},{"code_sha256_prefix":"e7fbc7a74a3457c7","entry":"load_vocab","repo":"ffhibnese/cpgc_vlp_universal_attacks","repo_kind":"official","path":"models/tokenization_bert.py","file_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks/blob/HEAD/models/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7fbc7a74a3457c7"}},{"code_sha256_prefix":"9a299fe5ae09e407","entry":"uniq","repo":"ffhibnese/cpgc_vlp_universal_attacks","repo_kind":"official","path":"attention.py","file_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks/blob/HEAD/attention.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a299fe5ae09e407"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"ffhibnese/cpgc_vlp_universal_attacks","repo_kind":"official","path":"models/model_pretrain.py","file_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks/blob/HEAD/models/model_pretrain.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"26be70dca3249c0b","entry":"load_tf_weights_in_bert","repo":"ffhibnese/cpgc_vlp_universal_attacks","repo_kind":"official","path":"models/xbert.py","file_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks/blob/HEAD/models/xbert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"26be70dca3249c0b"}},{"code_sha256_prefix":"8a57c3e7d1cf36c7","entry":"pre_caption","repo":"ffhibnese/cpgc_vlp_universal_attacks","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/ffhibnese/cpgc_vlp_universal_attacks/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a57c3e7d1cf36c7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}