{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/better-safe-than-sorry-pre-training-clip","title":"Better Safe than Sorry: Pre-training CLIP against Targeted Data Poisoning and Backdoor Attacks","arxiv_id":"2310.05862","date":"2023-10-05","proceeding":null,"authors":["Wenhan Yang","Jingdong Gao","Baharan Mirzasoleiman"],"abstract":"Contrastive Language-Image Pre-training (CLIP) on large image-caption datasets has achieved remarkable success in zero-shot classification and enabled transferability to new domains. However, CLIP is extremely more vulnerable to targeted data poisoning and backdoor attacks, compared to supervised learning. Perhaps surprisingly, poisoning 0.0001% of CLIP pre-training data is enough to make targeted data poisoning attacks successful. This is four orders of magnitude smaller than what is required to poison supervised models. Despite this vulnerability, existing methods are very limited in defending CLIP models during pre-training. In this work, we propose a strong defense, SAFECLIP, to safely pre-train CLIP against targeted data poisoning and backdoor attacks. SAFECLIP warms up the model by applying unimodal contrastive learning (CL) on image and text modalities separately. Then, it divides the data into safe and risky sets, by applying a Gaussian Mixture Model to the cosine similarity of image-caption pair representations. SAFECLIP pre-trains the model by applying the CLIP loss to the safe set and applying unimodal CL to image and text modalities of the risky set separately. By gradually increasing the size of the safe set during pre-training, SAFECLIP effectively breaks targeted data poisoning and backdoor attacks without harming the CLIP performance. Our extensive experiments on CC3M, Visual Genome, and MSCOCO demonstrate that SAFECLIP significantly reduces the success rate of targeted data poisoning attacks from 93.75% to 0% and that of various backdoor attacks from up to 100% to 0%, without harming CLIP's performance.","url_abs":"https://arxiv.org/abs/2310.05862v2","url_pdf":"https://arxiv.org/pdf/2310.05862v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"better-safe-than-sorry-pre-training-clip","repo_url":"https://github.com/bigml-cs-ucla/safeclip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"data-poisoning","task_name":"Data Poisoning"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.05862","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.05862"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bigml-cs-ucla/safeclip","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran":4,"unverified":5},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"d36be288b124bd8b","entry":"apply","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d36be288b124bd8b"}},{"code_sha256_prefix":"18af334ee213157a","entry":"calcualte_num_batches","repo":"bigml-cs-ucla/safeclip","repo_kind":"official","path":"SafeCLIP/src/scheduler.py","file_url":"https://github.com/bigml-cs-ucla/safeclip/blob/HEAD/SafeCLIP/src/scheduler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"18af334ee213157a"}},{"code_sha256_prefix":"757591aac112e98e","entry":"cosine_scheduler","repo":"bigml-cs-ucla/safeclip","repo_kind":"official","path":"SafeCLIP/src/scheduler.py","file_url":"https://github.com/bigml-cs-ucla/safeclip/blob/HEAD/SafeCLIP/src/scheduler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"757591aac112e98e"}},{"code_sha256_prefix":"ce27383ada7b655d","entry":"fit_gmm_to_cos_sim","repo":"bigml-cs-ucla/safeclip","repo_kind":"official","path":"SafeCLIP/src/samplers.py","file_url":"https://github.com/bigml-cs-ucla/safeclip/blob/HEAD/SafeCLIP/src/samplers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce27383ada7b655d"}},{"code_sha256_prefix":"75f67d27c89765cd","entry":"get_logger","repo":"bigml-cs-ucla/safeclip","repo_kind":"official","path":"SafeCLIP/src/logger.py","file_url":"https://github.com/bigml-cs-ucla/safeclip/blob/HEAD/SafeCLIP/src/logger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"75f67d27c89765cd"}},{"code_sha256_prefix":"27b459903dd9ce81","entry":"build","repo":"bigml-cs-ucla/safeclip","repo_kind":"official","path":"SafeCLIP/pkgs/openai/model.py","file_url":"https://github.com/bigml-cs-ucla/safeclip/blob/HEAD/SafeCLIP/pkgs/openai/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27b459903dd9ce81"}},{"code_sha256_prefix":"37891dc545d9edac","entry":"get_linear_probe_metrics","repo":"bigml-cs-ucla/safeclip","repo_kind":"official","path":"SafeCLIP/src/evaluate.py","file_url":"https://github.com/bigml-cs-ucla/safeclip/blob/HEAD/SafeCLIP/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"37891dc545d9edac"}},{"code_sha256_prefix":"f7dab6ab4b974716","entry":"get_validation_metrics","repo":"bigml-cs-ucla/safeclip","repo_kind":"official","path":"SafeCLIP/src/evaluate.py","file_url":"https://github.com/bigml-cs-ucla/safeclip/blob/HEAD/SafeCLIP/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7dab6ab4b974716"}},{"code_sha256_prefix":"3273b611e02f5586","entry":"get_zeroshot_metrics","repo":"bigml-cs-ucla/safeclip","repo_kind":"official","path":"SafeCLIP/src/evaluate.py","file_url":"https://github.com/bigml-cs-ucla/safeclip/blob/HEAD/SafeCLIP/src/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3273b611e02f5586"}},{"code_sha256_prefix":"30c9fb7349669f71","entry":"multiprocess","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"30c9fb7349669f71"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}