{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revisiting-the-adversarial-robustness-of","title":"Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective","arxiv_id":"2404.19287","date":"2024-04-30","proceeding":null,"authors":["Wanqi Zhou","Shuanghao Bai","Danilo P. Mandic","Qibin Zhao","Badong Chen"],"abstract":"Pretrained vision-language models (VLMs) like CLIP exhibit exceptional generalization across diverse downstream tasks. While recent studies reveal their vulnerability to adversarial attacks, research to date has primarily focused on enhancing the robustness of image encoders against image-based attacks, with defenses against text-based and multimodal attacks remaining largely unexplored. To this end, this work presents the first comprehensive study on improving the adversarial robustness of VLMs against attacks targeting image, text, and multimodal inputs. This is achieved by proposing multimodal contrastive adversarial training (MMCoA). Such an approach strengthens the robustness of both image and text encoders by aligning the clean text embeddings with adversarial image embeddings, and adversarial text embeddings with clean image embeddings. The robustness of the proposed MMCoA is examined against existing defense methods over image, text, and multimodal attacks on the CLIP model. Extensive experiments on 15 datasets across two tasks reveal the characteristics of different adversarial defense methods under distinct distribution shifts and dataset complexities across the three attack types. This paves the way for a unified framework of adversarial robustness against different modality attacks, opening up new possibilities for securing VLMs against multimodal attacks. The code is available at https://github.com/ElleZWQ/MMCoA.git.","url_abs":"https://arxiv.org/abs/2404.19287v3","url_pdf":"https://arxiv.org/pdf/2404.19287v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"revisiting-the-adversarial-robustness-of","repo_url":"https://github.com/ellezwq/mmcoa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"adversarial-defense","task_name":"Adversarial Defense"},{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"adversarial-text","task_name":"Adversarial Text"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2404.19287","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.19287"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ellezwq/mmcoa","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran_fixture":3,"ran":7,"ran_honours":1,"unverified":8},"by_repo_kind":{"official":{"samples":22,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cf9ffa02a42184af","entry":"whitespace_tokenize","repo":"ellezwq/mmcoa","repo_kind":"official","path":"models/tokenization_bert.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/models/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf9ffa02a42184af"}},{"code_sha256_prefix":"b0f936d4d6ae3b8c","entry":"accuracy","repo":"ellezwq/mmcoa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0f936d4d6ae3b8c"}},{"code_sha256_prefix":"2e21f055fab549ee","entry":"calculate_accuracy","repo":"ellezwq/mmcoa","repo_kind":"official","path":"MMCoA.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/MMCoA.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e21f055fab549ee"}},{"code_sha256_prefix":"1f2c3eac81b5d4f4","entry":"clamp_by_l2","repo":"ellezwq/mmcoa","repo_kind":"official","path":"attack/imageAttack.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/attack/imageAttack.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f2c3eac81b5d4f4"}},{"code_sha256_prefix":"aa81d03f07735ab8","entry":"compute_acc","repo":"ellezwq/mmcoa","repo_kind":"official","path":"utils_2.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/utils_2.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa81d03f07735ab8"}},{"code_sha256_prefix":"ad12a494674d23fb","entry":"compute_n_params","repo":"ellezwq/mmcoa","repo_kind":"official","path":"utils_2.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/utils_2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad12a494674d23fb"}},{"code_sha256_prefix":"81b93fc3c40a9fa0","entry":"equal_normalize","repo":"ellezwq/mmcoa","repo_kind":"official","path":"attack/multimodalAttack.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/attack/multimodalAttack.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81b93fc3c40a9fa0"}},{"code_sha256_prefix":"263288717b164fe4","entry":"fixed_patch","repo":"ellezwq/mmcoa","repo_kind":"official","path":"models/prompters.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/models/prompters.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"263288717b164fe4"}},{"code_sha256_prefix":"d121638ef1dea109","entry":"get_key","repo":"ellezwq/mmcoa","repo_kind":"official","path":"utils_2.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/utils_2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d121638ef1dea109"}},{"code_sha256_prefix":"5e1f60c820f3f710","entry":"gkern","repo":"ellezwq/mmcoa","repo_kind":"official","path":"attack/imageAttack.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/attack/imageAttack.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e1f60c820f3f710"}},{"code_sha256_prefix":"e7fbc7a74a3457c7","entry":"load_vocab","repo":"ellezwq/mmcoa","repo_kind":"official","path":"models/tokenization_bert.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/models/tokenization_bert.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7fbc7a74a3457c7"}},{"code_sha256_prefix":"ed6b8fbf59cfc213","entry":"padding","repo":"ellezwq/mmcoa","repo_kind":"official","path":"models/prompters.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/models/prompters.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed6b8fbf59cfc213"}},{"code_sha256_prefix":"45669d4d47a3a910","entry":"random_init","repo":"ellezwq/mmcoa","repo_kind":"official","path":"attack/imageAttack.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/attack/imageAttack.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45669d4d47a3a910"}},{"code_sha256_prefix":"d22c609a1ecad7e5","entry":"random_patch","repo":"ellezwq/mmcoa","repo_kind":"official","path":"models/prompters.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/models/prompters.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d22c609a1ecad7e5"}},{"code_sha256_prefix":"7c90002916e1665a","entry":"attack_pgd","repo":"ellezwq/mmcoa","repo_kind":"official","path":"attacks.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/attacks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c90002916e1665a"}},{"code_sha256_prefix":"8a93e041134b597a","entry":"clamp","repo":"ellezwq/mmcoa","repo_kind":"official","path":"attacks.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/attacks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a93e041134b597a"}},{"code_sha256_prefix":"932d44bbf8bd0e3f","entry":"cosine_lr","repo":"ellezwq/mmcoa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"932d44bbf8bd0e3f"}},{"code_sha256_prefix":"cad29a7a417c68d3","entry":"create_logits","repo":"ellezwq/mmcoa","repo_kind":"official","path":"models/model.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/models/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cad29a7a417c68d3"}},{"code_sha256_prefix":"0b855e55aafa281e","entry":"get_bpe_substitues","repo":"ellezwq/mmcoa","repo_kind":"official","path":"attack/bert_attack.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/attack/bert_attack.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b855e55aafa281e"}},{"code_sha256_prefix":"340f1e13ccb625f4","entry":"get_substitues","repo":"ellezwq/mmcoa","repo_kind":"official","path":"attack/bert_attack.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/attack/bert_attack.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"340f1e13ccb625f4"}},{"code_sha256_prefix":"9e585a3531b71996","entry":"load","repo":"ellezwq/mmcoa","repo_kind":"official","path":"replace/clip.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/replace/clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e585a3531b71996"}},{"code_sha256_prefix":"60ab7dcbb5498f88","entry":"refine_classname","repo":"ellezwq/mmcoa","repo_kind":"official","path":"utils.py","file_url":"https://github.com/ellezwq/mmcoa/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60ab7dcbb5498f88"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}