{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/set-level-guidance-attack-boosting","title":"Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models","arxiv_id":"2307.14061","date":"2023-07-26","proceeding":"ICCV 2023 1","authors":["Dong Lu","Zhiqiang Wang","Teng Wang","Weili Guan","Hongchang Gao","Feng Zheng"],"abstract":"Vision-language pre-training (VLP) models have shown vulnerability to adversarial examples in multimodal tasks. Furthermore, malicious adversaries can be deliberately transferred to attack other black-box models. However, existing work has mainly focused on investigating white-box attacks. In this paper, we present the first study to investigate the adversarial transferability of recent VLP models. We observe that existing methods exhibit much lower transferability, compared to the strong attack performance in white-box settings. The transferability degradation is partly caused by the under-utilization of cross-modal interactions. Particularly, unlike unimodal learning, VLP models rely heavily on cross-modal interactions and the multimodal alignments are many-to-many, e.g., an image can be described in various natural languages. To this end, we propose a highly transferable Set-level Guidance Attack (SGA) that thoroughly leverages modality interactions and incorporates alignment-preserving augmentation with cross-modal guidance. Experimental results demonstrate that SGA could generate adversarial examples that can strongly transfer across different VLP models on multiple downstream vision-language tasks. On image-text retrieval, SGA significantly enhances the attack success rate for transfer attacks from ALBEF to TCL by a large margin (at least 9.78% and up to 30.21%), compared to the state-of-the-art.","url_abs":"https://arxiv.org/abs/2307.14061v1","url_pdf":"https://arxiv.org/pdf/2307.14061v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"set-level-guidance-attack-boosting","repo_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-text-retrieval","task_name":"Image-text Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"}],"methods":[{"method_slug":"albef","method_name":"ALBEF"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.14061","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.14061"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/Zoky-2020/SGA","reach":null}],"summary":{"ran_draft_wrong":4,"ran":2,"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":11,"ran":6,"repositories":1},"found_in_text":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cf9ffa02a42184af","entry":"whitespace_tokenize","repo":"Zoky-2020/Set-level_Guidance_Attack","repo_kind":"official","path":"models/tokenization_bert.py","file_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack/blob/HEAD/models/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf9ffa02a42184af"}},{"code_sha256_prefix":"054e91659c422b0e","entry":"SGAttacker","repo":"Zoky-2020/SGA","repo_kind":"found_in_text","path":"attacker.py","file_url":"https://github.com/Zoky-2020/SGA/blob/HEAD/attacker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"054e91659c422b0e"}},{"code_sha256_prefix":"aa81d03f07735ab8","entry":"compute_acc","repo":"Zoky-2020/Set-level_Guidance_Attack","repo_kind":"official","path":"utils.py","file_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa81d03f07735ab8"}},{"code_sha256_prefix":"ad12a494674d23fb","entry":"compute_n_params","repo":"Zoky-2020/Set-level_Guidance_Attack","repo_kind":"official","path":"utils.py","file_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad12a494674d23fb"}},{"code_sha256_prefix":"c6ec173f19f5c34d","entry":"interpolate_pos_embed","repo":"Zoky-2020/Set-level_Guidance_Attack","repo_kind":"official","path":"models/vit.py","file_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack/blob/HEAD/models/vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6ec173f19f5c34d"}},{"code_sha256_prefix":"98ae1974cb453187","entry":"itm_eval","repo":"Zoky-2020/Set-level_Guidance_Attack","repo_kind":"official","path":"eval_albef2clip-vit_flickr.py","file_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack/blob/HEAD/eval_albef2clip-vit_flickr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98ae1974cb453187"}},{"code_sha256_prefix":"e7fbc7a74a3457c7","entry":"load_vocab","repo":"Zoky-2020/Set-level_Guidance_Attack","repo_kind":"official","path":"models/tokenization_bert.py","file_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack/blob/HEAD/models/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7fbc7a74a3457c7"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"Zoky-2020/Set-level_Guidance_Attack","repo_kind":"official","path":"models/model_pretrain.py","file_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack/blob/HEAD/models/model_pretrain.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"0b855e55aafa281e","entry":"get_bpe_substitues","repo":"Zoky-2020/Set-level_Guidance_Attack","repo_kind":"official","path":"attacker.py","file_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack/blob/HEAD/attacker.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b855e55aafa281e"}},{"code_sha256_prefix":"340f1e13ccb625f4","entry":"get_substitues","repo":"Zoky-2020/Set-level_Guidance_Attack","repo_kind":"official","path":"attacker.py","file_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack/blob/HEAD/attacker.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"340f1e13ccb625f4"}},{"code_sha256_prefix":"26be70dca3249c0b","entry":"load_tf_weights_in_bert","repo":"Zoky-2020/Set-level_Guidance_Attack","repo_kind":"official","path":"models/xbert.py","file_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack/blob/HEAD/models/xbert.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26be70dca3249c0b"}},{"code_sha256_prefix":"8a57c3e7d1cf36c7","entry":"pre_caption","repo":"Zoky-2020/Set-level_Guidance_Attack","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/Zoky-2020/Set-level_Guidance_Attack/blob/HEAD/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a57c3e7d1cf36c7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}