{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/badclip-dual-embedding-guided-backdoor-attack","title":"BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning","arxiv_id":"2311.12075","date":"2023-11-20","proceeding":"CVPR 2024 1","authors":["Siyuan Liang","Mingli Zhu","Aishan Liu","Baoyuan Wu","Xiaochun Cao","Ee-Chien Chang"],"abstract":"Studying backdoor attacks is valuable for model copyright protection and enhancing defenses. While existing backdoor attacks have successfully infected multimodal contrastive learning models such as CLIP, they can be easily countered by specialized backdoor defenses for MCL models. This paper reveals the threats in this practical scenario that backdoor attacks can remain effective even after defenses and introduces the \\emph{\\toolns} attack, which is resistant to backdoor detection and model fine-tuning defenses. To achieve this, we draw motivations from the perspective of the Bayesian rule and propose a dual-embedding guided framework for backdoor attacks. Specifically, we ensure that visual trigger patterns approximate the textual target semantics in the embedding space, making it challenging to detect the subtle parameter variations induced by backdoor learning on such natural trigger patterns. Additionally, we optimize the visual trigger patterns to align the poisoned samples with target vision features in order to hinder the backdoor unlearning through clean fine-tuning. Extensive experiments demonstrate that our attack significantly outperforms state-of-the-art baselines (+45.3% ASR) in the presence of SoTA backdoor defenses, rendering these mitigation and detection strategies virtually ineffective. Furthermore, our approach effectively attacks some more rigorous scenarios like downstream tasks. We believe that this paper raises awareness regarding the potential threats associated with the practical application of multimodal contrastive learning and encourages the development of more robust defense mechanisms.","url_abs":"https://arxiv.org/abs/2311.12075v3","url_pdf":"https://arxiv.org/pdf/2311.12075v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"badclip-dual-embedding-guided-backdoor-attack","repo_url":"https://github.com/LiangSiyuan21/BadCLIP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"backdoor-attack","task_name":"Backdoor Attack"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.12075","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.12075"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LiangSiyuan21/BadCLIP","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":2,"ran_violates":1},"by_repo_kind":{"listed":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"77a3a0f7b9c19883","entry":"cosine_triplet_loss","repo":"LiangSiyuan21/BadCLIP","repo_kind":"listed","path":"src/embeding_optimize_patch.py","file_url":"https://github.com/LiangSiyuan21/BadCLIP/blob/HEAD/src/embeding_optimize_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77a3a0f7b9c19883"}},{"code_sha256_prefix":"56a77cf2cc05fc38","entry":"get_loss","repo":"LiangSiyuan21/BadCLIP","repo_kind":"listed","path":"src/embeding_optimize_patch.py","file_url":"https://github.com/LiangSiyuan21/BadCLIP/blob/HEAD/src/embeding_optimize_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56a77cf2cc05fc38"}},{"code_sha256_prefix":"d3d38079078a4f51","entry":"prepare_path_name","repo":"LiangSiyuan21/BadCLIP","repo_kind":"listed","path":"backdoor/create_backdoor_data.py","file_url":"https://github.com/LiangSiyuan21/BadCLIP/blob/HEAD/backdoor/create_backdoor_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3d38079078a4f51"}},{"code_sha256_prefix":"11cf7e14cfed92f1","entry":"triplet_loss","repo":"LiangSiyuan21/BadCLIP","repo_kind":"listed","path":"src/embeding_optimize_patch.py","file_url":"https://github.com/LiangSiyuan21/BadCLIP/blob/HEAD/src/embeding_optimize_patch.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11cf7e14cfed92f1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}