{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/clip-is-strong-enough-to-fight-back-test-time","title":"CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP","arxiv_id":"2503.03613","date":"2025-03-05","proceeding":"CVPR 2025 1","authors":["Songlong Xing","Zhengyu Zhao","Nicu Sebe"],"abstract":"Despite its prevalent use in image-text matching tasks in a zero-shot manner, CLIP has been shown to be highly vulnerable to adversarial perturbations added onto images. Recent studies propose to finetune the vision encoder of CLIP with adversarial samples generated on the fly, and show improved robustness against adversarial attacks on a spectrum of downstream datasets, a property termed as zero-shot robustness. In this paper, we show that malicious perturbations that seek to maximise the classification loss lead to `falsely stable' images, and propose to leverage the pre-trained vision encoder of CLIP to counterattack such adversarial images during inference to achieve robustness. Our paradigm is simple and training-free, providing the first method to defend CLIP from adversarial attacks at test time, which is orthogonal to existing methods aiming to boost zero-shot adversarial robustness of CLIP. We conduct experiments across 16 classification datasets, and demonstrate stable and consistent gains compared to test-time defence methods adapted from existing adversarial robustness studies that do not rely on external networks, without noticeably impairing performance on clean images. We also show that our paradigm can be employed on CLIP models that have been adversarially finetuned to further enhance their robustness at test time. Our code is available \\href{https://github.com/Sxing2/CLIP-Test-time-Counterattacks}{here}.","url_abs":"https://arxiv.org/abs/2503.03613v1","url_pdf":"https://arxiv.org/pdf/2503.03613v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"clip-is-strong-enough-to-fight-back-test-time","repo_url":"https://github.com/Sxing2/CLIP-Test-time-Counterattacks","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"},{"task_slug":"image-text-matching","task_name":"Image-text matching"},{"task_slug":"text-matching","task_name":"Text Matching"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.03613","atlas_url":"https://app.syntology.ai/?focus=2503.03613","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.03613"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Sxing2/CLIP-Test-time-Counterattacks","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"29a9e9c6629ed618","entry":"clip_img_preprocessing","repo":"Sxing2/CLIP-Test-time-Counterattacks","repo_kind":"official","path":"code/attacks.py","file_url":"https://github.com/Sxing2/CLIP-Test-time-Counterattacks/blob/HEAD/code/attacks.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"29a9e9c6629ed618"}},{"code_sha256_prefix":"dd42cbeb18af19eb","entry":"normalize","repo":"Sxing2/CLIP-Test-time-Counterattacks","repo_kind":"official","path":"code/attacks.py","file_url":"https://github.com/Sxing2/CLIP-Test-time-Counterattacks/blob/HEAD/code/attacks.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd42cbeb18af19eb"}},{"code_sha256_prefix":"f5dcab1b5724f573","entry":"attack_unlabelled_opp","repo":"Sxing2/CLIP-Test-time-Counterattacks","repo_kind":"official","path":"code/attacks.py","file_url":"https://github.com/Sxing2/CLIP-Test-time-Counterattacks/blob/HEAD/code/attacks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f5dcab1b5724f573"}},{"code_sha256_prefix":"8a93e041134b597a","entry":"clamp","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"8a93e041134b597a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}