{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adversarial-prompt-tuning-for-vision-language","title":"Adversarial Prompt Tuning for Vision-Language Models","arxiv_id":"2311.11261","date":"2023-11-19","proceeding":null,"authors":["Jiaming Zhang","Xingjun Ma","Xin Wang","Lingyu Qiu","Jiaqi Wang","Yu-Gang Jiang","Jitao Sang"],"abstract":"With the rapid advancement of multimodal learning, pre-trained Vision-Language Models (VLMs) such as CLIP have demonstrated remarkable capacities in bridging the gap between visual and language modalities. However, these models remain vulnerable to adversarial attacks, particularly in the image modality, presenting considerable security risks. This paper introduces Adversarial Prompt Tuning (AdvPT), a novel technique to enhance the adversarial robustness of image encoders in VLMs. AdvPT innovatively leverages learnable text prompts and aligns them with adversarial image embeddings, to address the vulnerabilities inherent in VLMs without the need for extensive parameter training or modification of the model architecture. We demonstrate that AdvPT improves resistance against white-box and black-box adversarial attacks and exhibits a synergistic effect when combined with existing image-processing-based defense techniques, further boosting defensive capabilities. Comprehensive experimental analyses provide insights into adversarial prompt tuning, a novel paradigm devoted to improving resistance to adversarial images through textual input modifications, paving the way for future robust multimodal learning research. These findings open up new possibilities for enhancing the security of VLMs. Our code is available at https://github.com/jiamingzhang94/Adversarial-Prompt-Tuning.","url_abs":"https://arxiv.org/abs/2311.11261v3","url_pdf":"https://arxiv.org/pdf/2311.11261v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adversarial-prompt-tuning-for-vision-language","repo_url":"https://github.com/jiamingzhang94/adversarial-prompt-tuning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.11261","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.11261"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiamingzhang94/adversarial-prompt-tuning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ba26afd892405335","entry":"compute_ci95","repo":"jiamingzhang94/adversarial-prompt-tuning","repo_kind":"official","path":"parse_test_res.py","file_url":"https://github.com/jiamingzhang94/adversarial-prompt-tuning/blob/HEAD/parse_test_res.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba26afd892405335"}},{"code_sha256_prefix":"419e1fe5de09cb00","entry":"wrap_model","repo":"jiamingzhang94/adversarial-prompt-tuning","repo_kind":"official","path":"black.py","file_url":"https://github.com/jiamingzhang94/adversarial-prompt-tuning/blob/HEAD/black.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"419e1fe5de09cb00"}},{"code_sha256_prefix":"39e6b23b55f376ea","entry":"build_model","repo":"jiamingzhang94/adversarial-prompt-tuning","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/jiamingzhang94/adversarial-prompt-tuning/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39e6b23b55f376ea"}},{"code_sha256_prefix":"2da7ec0975be872a","entry":"load","repo":"jiamingzhang94/adversarial-prompt-tuning","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/jiamingzhang94/adversarial-prompt-tuning/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2da7ec0975be872a"}},{"code_sha256_prefix":"f861f234105fa2aa","entry":"super_resolution","repo":"jiamingzhang94/adversarial-prompt-tuning","repo_kind":"official","path":"attack/purification.py","file_url":"https://github.com/jiamingzhang94/adversarial-prompt-tuning/blob/HEAD/attack/purification.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f861f234105fa2aa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}