{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/text-to-image-diffusion-models-can-be-easily","title":"Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning","arxiv_id":"2305.04175","date":"2023-05-07","proceeding":null,"authors":["Shengfang Zhai","Yinpeng Dong","Qingni Shen","Shi Pu","Yuejian Fang","Hang Su"],"abstract":"With the help of conditioning mechanisms, the state-of-the-art diffusion models have achieved tremendous success in guided image generation, particularly in text-to-image synthesis. To gain a better understanding of the training process and potential risks of text-to-image synthesis, we perform a systematic investigation of backdoor attack on text-to-image diffusion models and propose BadT2I, a general multimodal backdoor attack framework that tampers with image synthesis in diverse semantic levels. Specifically, we perform backdoor attacks on three levels of the vision semantics: Pixel-Backdoor, Object-Backdoor and Style-Backdoor. By utilizing a regularization loss, our methods efficiently inject backdoors into a large-scale text-to-image diffusion model while preserving its utility with benign inputs. We conduct empirical experiments on Stable Diffusion, the widely-used text-to-image diffusion model, demonstrating that the large-scale diffusion model can be easily backdoored within a few fine-tuning steps. We conduct additional experiments to explore the impact of different types of textual triggers, as well as the backdoor persistence during further training, providing insights for the development of backdoor defense methods. Besides, our investigation may contribute to the copyright protection of text-to-image models in the future.","url_abs":"https://arxiv.org/abs/2305.04175v2","url_pdf":"https://arxiv.org/pdf/2305.04175v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"text-to-image-diffusion-models-can-be-easily","repo_url":"https://github.com/sf-zhai/badt2i","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"backdoor-attack","task_name":"Backdoor Attack"},{"task_slug":"data-poisoning","task_name":"Data Poisoning"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"backdoor-defense","task_name":"backdoor defense"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.04175","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.04175"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sf-zhai/badt2i","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c9ec7a876028024d","entry":"create_output_folder","repo":"sf-zhai/badt2i","repo_kind":"official","path":"evaluation/img_gen.py","file_url":"https://github.com/sf-zhai/badt2i/blob/HEAD/evaluation/img_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9ec7a876028024d"}},{"code_sha256_prefix":"b10638497194fcfc","entry":"generate_latents","repo":"sf-zhai/badt2i","repo_kind":"official","path":"evaluation/img_gen.py","file_url":"https://github.com/sf-zhai/badt2i/blob/HEAD/evaluation/img_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b10638497194fcfc"}},{"code_sha256_prefix":"8dacafd52cd06522","entry":"get_full_repo_name","repo":"sf-zhai/badt2i","repo_kind":"official","path":"badt2i_object.py","file_url":"https://github.com/sf-zhai/badt2i/blob/HEAD/badt2i_object.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8dacafd52cd06522"}},{"code_sha256_prefix":"bbd2ac7ff643d1cf","entry":"load_prompts","repo":"sf-zhai/badt2i","repo_kind":"official","path":"evaluation/img_gen.py","file_url":"https://github.com/sf-zhai/badt2i/blob/HEAD/evaluation/img_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bbd2ac7ff643d1cf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}