{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/few-shot-adversarial-prompt-learning-on","title":"Few-Shot Adversarial Prompt Learning on Vision-Language Models","arxiv_id":"2403.14774","date":"2024-03-21","proceeding":null,"authors":["Yiwei Zhou","Xiaobo Xia","Zhiwei Lin","Bo Han","Tongliang Liu"],"abstract":"The vulnerability of deep neural networks to imperceptible adversarial perturbations has attracted widespread attention. Inspired by the success of vision-language foundation models, previous efforts achieved zero-shot adversarial robustness by aligning adversarial visual features with text supervision. However, in practice, they are still unsatisfactory due to several issues, including heavy adaptation cost, suboptimal text supervision, and uncontrolled natural generalization capacity. In this paper, to address these issues, we propose a few-shot adversarial prompt framework where adapting input sequences with limited data makes significant adversarial robustness improvement. Specifically, we achieve this by providing adversarially correlated text supervision that is end-to-end learned from adversarial examples. We also propose a novel training objective that enhances the consistency of multi-modal features while encourages differentiated uni-modal features between natural and adversarial examples. The proposed framework gives access to learn adversarial text supervision, which provides superior cross-modal adversarial alignment and matches state-of-the-art zero-shot adversarial robustness with only 1% training data. Code is available at: https://github.com/lionel-w2/FAP.","url_abs":"https://arxiv.org/abs/2403.14774v2","url_pdf":"https://arxiv.org/pdf/2403.14774v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"few-shot-adversarial-prompt-learning-on","repo_url":"https://github.com/lionel-w2/fap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"adversarial-text","task_name":"Adversarial Text"},{"task_slug":"prompt-learning","task_name":"Prompt Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.14774","atlas_url":"https://app.syntology.ai/?focus=2403.14774","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.14774"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/lionel-w2/FAP","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lionel-w2/fap","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":5,"unverified":4},"by_repo_kind":{"official":{"samples":9,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"lionel-w2/FAP","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/lionel-w2/FAP/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"3e53cbc1ee35936a","entry":"calculate_cosine_similarity","repo":"lionel-w2/fap","repo_kind":"official","path":"trainers/fap.py","file_url":"https://github.com/lionel-w2/fap/blob/HEAD/trainers/fap.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e53cbc1ee35936a"}},{"code_sha256_prefix":"6561a62f97b03294","entry":"calculate_elementwise_kl_div","repo":"lionel-w2/fap","repo_kind":"official","path":"trainers/fap.py","file_url":"https://github.com/lionel-w2/fap/blob/HEAD/trainers/fap.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6561a62f97b03294"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"lionel-w2/FAP","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/lionel-w2/FAP/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"lionel-w2/FAP","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/lionel-w2/FAP/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"43334d2162325a01","entry":"attack_pgd","repo":"lionel-w2/FAP","repo_kind":"official","path":"attack/pgd.py","file_url":"https://github.com/lionel-w2/FAP/blob/HEAD/attack/pgd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43334d2162325a01"}},{"code_sha256_prefix":"48ac2bd4dcbfb717","entry":"build_model","repo":"lionel-w2/FAP","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/lionel-w2/FAP/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48ac2bd4dcbfb717"}},{"code_sha256_prefix":"8a93e041134b597a","entry":"clamp","repo":"lionel-w2/FAP","repo_kind":"official","path":"attack/pgd.py","file_url":"https://github.com/lionel-w2/FAP/blob/HEAD/attack/pgd.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a93e041134b597a"}},{"code_sha256_prefix":"fbf8c0143d9c48e3","entry":"load","repo":"lionel-w2/FAP","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/lionel-w2/FAP/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbf8c0143d9c48e3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}