{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prompting4debugging-red-teaming-text-to-image","title":"Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts","arxiv_id":"2309.06135","date":"2023-09-12","proceeding":null,"authors":["Zhi-Yi Chin","Chieh-Ming Jiang","Ching-Chun Huang","Pin-Yu Chen","Wei-Chen Chiu"],"abstract":"Text-to-image diffusion models, e.g. Stable Diffusion (SD), lately have shown remarkable ability in high-quality content generation, and become one of the representatives for the recent wave of transformative AI. Nevertheless, such advance comes with an intensifying concern about the misuse of this generative technology, especially for producing copyrighted or NSFW (i.e. not safe for work) images. Although efforts have been made to filter inappropriate images/prompts or remove undesirable concepts/styles via model fine-tuning, the reliability of these safety mechanisms against diversified problematic prompts remains largely unexplored. In this work, we propose Prompting4Debugging (P4D) as a debugging and red-teaming tool that automatically finds problematic prompts for diffusion models to test the reliability of a deployed safety mechanism. We demonstrate the efficacy of our P4D tool in uncovering new vulnerabilities of SD models with safety mechanisms. Particularly, our result shows that around half of prompts in existing safe prompting benchmarks which were originally considered \"safe\" can actually be manipulated to bypass many deployed safety mechanisms, including concept removal, negative prompt, and safety guidance. Our findings suggest that, without comprehensive testing, the evaluations on limited safe prompting benchmarks can lead to a false sense of safety for text-to-image models.","url_abs":"https://arxiv.org/abs/2309.06135v2","url_pdf":"https://arxiv.org/pdf/2309.06135v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prompting4debugging-red-teaming-text-to-image","repo_url":"https://github.com/joycenerd/p4d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"red-teaming","task_name":"Red Teaming"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"trustable-and-focussed-llm-generated-content","task_name":"trustable and focussed LLM generated content"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[{"slug":"p4d-prompts","name":"P4D prompts","full_name":"P4D universal jailbreaking prompt for T2I models"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.06135","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.06135"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joycenerd/p4d","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"community":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ee7c934cb3b5c4f9","entry":"load_dataset","repo":"zhiyichin/p4d","repo_kind":"community","path":"run_p4dn.py","file_url":"https://github.com/zhiyichin/p4d/blob/HEAD/run_p4dn.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee7c934cb3b5c4f9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}