{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cleanclip-mitigating-data-poisoning-attacks","title":"CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning","arxiv_id":"2303.03323","date":"2023-03-06","proceeding":"ICCV 2023 1","authors":["Hritik Bansal","Nishad Singhi","Yu Yang","Fan Yin","Aditya Grover","Kai-Wei Chang"],"abstract":"Multimodal contrastive pretraining has been used to train multimodal representation models, such as CLIP, on large amounts of paired image-text data. However, previous studies have revealed that such models are vulnerable to backdoor attacks. Specifically, when trained on backdoored examples, CLIP learns spurious correlations between the embedded backdoor trigger and the target label, aligning their representations in the joint embedding space. Injecting even a small number of poisoned examples, such as 75 examples in 3 million pretraining data, can significantly manipulate the model's behavior, making it difficult to detect or unlearn such correlations. To address this issue, we propose CleanCLIP, a finetuning framework that weakens the learned spurious associations introduced by backdoor attacks by independently re-aligning the representations for individual modalities. We demonstrate that unsupervised finetuning using a combination of multimodal contrastive and unimodal self-supervised objectives for individual modalities can significantly reduce the impact of the backdoor attack. Additionally, we show that supervised finetuning on task-specific labeled image data removes the backdoor trigger from the CLIP vision encoder. We show empirically that CleanCLIP maintains model performance on benign examples while erasing a range of backdoor attacks on multimodal contrastive learning. The code and checkpoints are available at https://github.com/nishadsinghi/CleanCLIP.","url_abs":"https://arxiv.org/abs/2303.03323v3","url_pdf":"https://arxiv.org/pdf/2303.03323v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cleanclip-mitigating-data-poisoning-attacks","repo_url":"https://github.com/nishadsinghi/cleanclip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"backdoor-attack","task_name":"Backdoor Attack"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"data-poisoning","task_name":"Data Poisoning"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.03323","atlas_url":"https://app.syntology.ai/?focus=2303.03323","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.03323"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nishadsinghi/cleanclip","reach":null}],"summary":{"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fabf8a8a0fd3b8e4","entry":"_augment_image","repo":"nishadsinghi/cleanclip","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/nishadsinghi/cleanclip/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fabf8a8a0fd3b8e4"}},{"code_sha256_prefix":"e7e095e74efcd040","entry":"ImageCaptionDataset","repo":"nishadsinghi/cleanclip","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/nishadsinghi/cleanclip/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e095e74efcd040"}},{"code_sha256_prefix":"dfc7c7c26c450b84","entry":"_augment_text","repo":"nishadsinghi/cleanclip","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/nishadsinghi/cleanclip/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfc7c7c26c450b84"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}