{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/robust-clip-unsupervised-adversarial-fine","title":"Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models","arxiv_id":"2402.12336","date":"2024-02-19","proceeding":null,"authors":["Christian Schlarmann","Naman Deep Singh","Francesco Croce","Matthias Hein"],"abstract":"Multi-modal foundation models like OpenFlamingo, LLaVA, and GPT-4 are increasingly used for various real-world tasks. Prior work has shown that these models are highly vulnerable to adversarial attacks on the vision modality. These attacks can be leveraged to spread fake information or defraud users, and thus pose a significant risk, which makes the robustness of large multi-modal foundation models a pressing problem. The CLIP model, or one of its variants, is used as a frozen vision encoder in many large vision-language models (LVLMs), e.g. LLaVA and OpenFlamingo. We propose an unsupervised adversarial fine-tuning scheme to obtain a robust CLIP vision encoder, which yields robustness on all vision down-stream tasks (LVLMs, zero-shot classification) that rely on CLIP. In particular, we show that stealth-attacks on users of LVLMs by a malicious third party providing manipulated images are no longer possible once one replaces the original CLIP model with our robust one. No retraining or fine-tuning of the down-stream LVLMs is required. The code and robust models are available at https://github.com/chs20/RobustVLM","url_abs":"https://arxiv.org/abs/2402.12336v2","url_pdf":"https://arxiv.org/pdf/2402.12336v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"robust-clip-unsupervised-adversarial-fine","repo_url":"https://github.com/chs20/robustvlm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-defense","task_name":"Adversarial Defense"},{"task_slug":"multimodal-deep-learning","task_name":"Multimodal Deep Learning"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.12336","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.12336"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chs20/robustvlm","reach":null}],"summary":{"ran":1,"ran_draft_wrong":3,"ran_fixture":3,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5bfccc77a86a50fb","entry":"ComputeLossWrapper","repo":"chs20/robustvlm","repo_kind":"official","path":"train/adversarial_training_clip.py","file_url":"https://github.com/chs20/robustvlm/blob/HEAD/train/adversarial_training_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5bfccc77a86a50fb"}},{"code_sha256_prefix":"1f2be2216dc3f7cd","entry":"compute_acc","repo":"chs20/robustvlm","repo_kind":"official","path":"train/adversarial_training_clip.py","file_url":"https://github.com/chs20/robustvlm/blob/HEAD/train/adversarial_training_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1f2be2216dc3f7cd"}},{"code_sha256_prefix":"75f68885e2ff2684","entry":"compute_loss","repo":"chs20/robustvlm","repo_kind":"official","path":"train/adversarial_training_clip.py","file_url":"https://github.com/chs20/robustvlm/blob/HEAD/train/adversarial_training_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75f68885e2ff2684"}},{"code_sha256_prefix":"b03f0274dabbb8c0","entry":"compute_loss","repo":"chs20/robustvlm","repo_kind":"official","path":"train/adversarial_training_clip.py","file_url":"https://github.com/chs20/robustvlm/blob/HEAD/train/adversarial_training_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b03f0274dabbb8c0"}},{"code_sha256_prefix":"68d0f41190a26308","entry":"get_query_set","repo":"chs20/robustvlm","repo_kind":"official","path":"vlm_eval/run_evaluation.py","file_url":"https://github.com/chs20/robustvlm/blob/HEAD/vlm_eval/run_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68d0f41190a26308"}},{"code_sha256_prefix":"63ec736a72a6cb74","entry":"get_random_indices","repo":"chs20/robustvlm","repo_kind":"official","path":"vlm_eval/run_evaluation.py","file_url":"https://github.com/chs20/robustvlm/blob/HEAD/vlm_eval/run_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63ec736a72a6cb74"}},{"code_sha256_prefix":"7062c33b2cc96026","entry":"pgd","repo":"chs20/robustvlm","repo_kind":"official","path":"train/adversarial_training_clip.py","file_url":"https://github.com/chs20/robustvlm/blob/HEAD/train/adversarial_training_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7062c33b2cc96026"}},{"code_sha256_prefix":"1707d13b9dc6cb32","entry":"train_one_epoch","repo":"chs20/robustvlm","repo_kind":"official","path":"train/adversarial_training_clip.py","file_url":"https://github.com/chs20/robustvlm/blob/HEAD/train/adversarial_training_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1707d13b9dc6cb32"}},{"code_sha256_prefix":"04cf221e803c96d7","entry":"unwrap_model","repo":"chs20/robustvlm","repo_kind":"official","path":"train/adversarial_training_clip.py","file_url":"https://github.com/chs20/robustvlm/blob/HEAD/train/adversarial_training_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04cf221e803c96d7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}