{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/semantic-aligned-adversarial-evolution","title":"Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attack","arxiv_id":"2411.02669","date":"2024-11-04","proceeding":null,"authors":["Xiaojun Jia","Sensen Gao","Qing Guo","Ke Ma","Yihao Huang","Simeng Qin","Yang Liu","Ivor Tsang Fellow","Xiaochun Cao"],"abstract":"Vision-language pre-training (VLP) models excel at interpreting both images and text but remain vulnerable to multimodal adversarial examples (AEs). Advancing the generation of transferable AEs, which succeed across unseen models, is key to developing more robust and practical VLP models. Previous approaches augment image-text pairs to enhance diversity within the adversarial example generation process, aiming to improve transferability by expanding the contrast space of image-text features. However, these methods focus solely on diversity around the current AEs, yielding limited gains in transferability. To address this issue, we propose to increase the diversity of AEs by leveraging the intersection regions along the adversarial trajectory during optimization. Specifically, we propose sampling from adversarial evolution triangles composed of clean, historical, and current adversarial examples to enhance adversarial diversity. We provide a theoretical analysis to demonstrate the effectiveness of the proposed adversarial evolution triangle. Moreover, we find that redundant inactive dimensions can dominate similarity calculations, distorting feature matching and making AEs model-dependent with reduced transferability. Hence, we propose to generate AEs in the semantic image-text feature contrast space, which can project the original feature space into a semantic corpus subspace. The proposed semantic-aligned subspace can reduce the image feature redundancy, thereby improving adversarial transferability. Extensive experiments across different datasets and models demonstrate that the proposed method can effectively improve adversarial transferability and outperform state-of-the-art adversarial attack methods. The code is released at https://github.com/jiaxiaojunQAQ/SA-AET.","url_abs":"https://arxiv.org/abs/2411.02669v1","url_pdf":"https://arxiv.org/pdf/2411.02669v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"semantic-aligned-adversarial-evolution","repo_url":"https://github.com/jiaxiaojunqaq/sa-aet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"semantic-aligned-adversarial-evolution","repo_url":"https://github.com/sensengao/vlptransferattack","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"adversarial-attack","task_name":"Adversarial Attack"},{"task_slug":"diversity","task_name":"Diversity"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.02669","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.02669"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiaxiaojunqaq/sa-aet","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sensengao/vlptransferattack","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":4,"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cf9ffa02a42184af","entry":"whitespace_tokenize","repo":"jiaxiaojunqaq/sa-aet","repo_kind":"official","path":"models/tokenization_bert.py","file_url":"https://github.com/jiaxiaojunqaq/sa-aet/blob/HEAD/models/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf9ffa02a42184af"}},{"code_sha256_prefix":"aa81d03f07735ab8","entry":"compute_acc","repo":"jiaxiaojunqaq/sa-aet","repo_kind":"official","path":"utils.py","file_url":"https://github.com/jiaxiaojunqaq/sa-aet/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa81d03f07735ab8"}},{"code_sha256_prefix":"ad12a494674d23fb","entry":"compute_n_params","repo":"jiaxiaojunqaq/sa-aet","repo_kind":"official","path":"utils.py","file_url":"https://github.com/jiaxiaojunqaq/sa-aet/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad12a494674d23fb"}},{"code_sha256_prefix":"c6ec173f19f5c34d","entry":"interpolate_pos_embed","repo":"jiaxiaojunqaq/sa-aet","repo_kind":"official","path":"models/vit.py","file_url":"https://github.com/jiaxiaojunqaq/sa-aet/blob/HEAD/models/vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6ec173f19f5c34d"}},{"code_sha256_prefix":"e7fbc7a74a3457c7","entry":"load_vocab","repo":"jiaxiaojunqaq/sa-aet","repo_kind":"official","path":"models/tokenization_bert.py","file_url":"https://github.com/jiaxiaojunqaq/sa-aet/blob/HEAD/models/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7fbc7a74a3457c7"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"jiaxiaojunqaq/sa-aet","repo_kind":"official","path":"models/model_pretrain.py","file_url":"https://github.com/jiaxiaojunqaq/sa-aet/blob/HEAD/models/model_pretrain.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"0b855e55aafa281e","entry":"get_bpe_substitues","repo":"jiaxiaojunqaq/sa-aet","repo_kind":"official","path":"SA_AET.py","file_url":"https://github.com/jiaxiaojunqaq/sa-aet/blob/HEAD/SA_AET.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b855e55aafa281e"}},{"code_sha256_prefix":"340f1e13ccb625f4","entry":"get_substitues","repo":"jiaxiaojunqaq/sa-aet","repo_kind":"official","path":"SA_AET.py","file_url":"https://github.com/jiaxiaojunqaq/sa-aet/blob/HEAD/SA_AET.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"340f1e13ccb625f4"}},{"code_sha256_prefix":"bd53e33a4eefdec0","entry":"load","repo":"sensengao/vlptransferattack","repo_kind":"listed","path":"models/clip_model/clip.py","file_url":"https://github.com/sensengao/vlptransferattack/blob/HEAD/models/clip_model/clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd53e33a4eefdec0"}},{"code_sha256_prefix":"26be70dca3249c0b","entry":"load_tf_weights_in_bert","repo":"jiaxiaojunqaq/sa-aet","repo_kind":"official","path":"models/xbert.py","file_url":"https://github.com/jiaxiaojunqaq/sa-aet/blob/HEAD/models/xbert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26be70dca3249c0b"}},{"code_sha256_prefix":"8a57c3e7d1cf36c7","entry":"pre_caption","repo":"jiaxiaojunqaq/sa-aet","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/jiaxiaojunqaq/sa-aet/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a57c3e7d1cf36c7"}},{"code_sha256_prefix":"6c0d9b91effb3d4c","entry":"toImage","repo":"jiaxiaojunqaq/sa-aet","repo_kind":"official","path":"eval_AET.py","file_url":"https://github.com/jiaxiaojunqaq/sa-aet/blob/HEAD/eval_AET.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c0d9b91effb3d4c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}