{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/art-automatic-red-teaming-for-text-to-image","title":"ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users","arxiv_id":"2405.19360","date":"2024-05-24","proceeding":null,"authors":["Guanlin Li","Kangjie Chen","Shudong Zhang","Jie Zhang","Tianwei Zhang"],"abstract":"Large-scale pre-trained generative models are taking the world by storm, due to their abilities in generating creative content. Meanwhile, safeguards for these generative models are developed, to protect users' rights and safety, most of which are designed for large language models. Existing methods primarily focus on jailbreak and adversarial attacks, which mainly evaluate the model's safety under malicious prompts. Recent work found that manually crafted safe prompts can unintentionally trigger unsafe generations. To further systematically evaluate the safety risks of text-to-image models, we propose a novel Automatic Red-Teaming framework, ART. Our method leverages both vision language model and large language model to establish a connection between unsafe generations and their prompts, thereby more efficiently identifying the model's vulnerabilities. With our comprehensive experiments, we reveal the toxicity of the popular open-source text-to-image models. The experiments also validate the effectiveness, adaptability, and great diversity of ART. Additionally, we introduce three large-scale red-teaming datasets for studying the safety risks associated with text-to-image models. Datasets and models can be found in https://github.com/GuanlinLee/ART.","url_abs":"https://arxiv.org/abs/2405.19360v3","url_pdf":"https://arxiv.org/pdf/2405.19360v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"art-automatic-red-teaming-for-text-to-image","repo_url":"https://github.com/guanlinlee/art","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"red-teaming","task_name":"Red Teaming"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.19360","atlas_url":"https://app.syntology.ai/?focus=2405.19360","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.19360"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/GuanlinLee/ART","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/guanlinlee/art","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"ran_honours":2,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":12,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2edce6f7b993b57d","entry":"build_custom_prompt","repo":"GuanlinLee/ART","repo_kind":"official","path":"prompt_utils_for_llamaguard.py","file_url":"https://github.com/GuanlinLee/ART/blob/HEAD/prompt_utils_for_llamaguard.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2edce6f7b993b57d"}},{"code_sha256_prefix":"36902b4e02b9c5b5","entry":"create_conversation","repo":"GuanlinLee/ART","repo_kind":"official","path":"prompt_utils_for_llamaguard.py","file_url":"https://github.com/GuanlinLee/ART/blob/HEAD/prompt_utils_for_llamaguard.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"36902b4e02b9c5b5"}},{"code_sha256_prefix":"99d0987d1d2adf13","entry":"get_img_detect_result","repo":"GuanlinLee/ART","repo_kind":"official","path":"summarize_results.py","file_url":"https://github.com/GuanlinLee/ART/blob/HEAD/summarize_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99d0987d1d2adf13"}},{"code_sha256_prefix":"cc03fc7ad96b8a2f","entry":"get_prompt_detect_result","repo":"GuanlinLee/ART","repo_kind":"official","path":"summarize_results.py","file_url":"https://github.com/GuanlinLee/ART/blob/HEAD/summarize_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc03fc7ad96b8a2f"}},{"code_sha256_prefix":"4ccf94a830b3dbbf","entry":"infererence_sd","repo":"guanlinlee/art","repo_kind":"official","path":"generate_images.py","file_url":"https://github.com/guanlinlee/art/blob/HEAD/generate_images.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ccf94a830b3dbbf"}},{"code_sha256_prefix":"536c088dcbe9fc5d","entry":"load_images","repo":"guanlinlee/art","repo_kind":"official","path":"generate_images.py","file_url":"https://github.com/guanlinlee/art/blob/HEAD/generate_images.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"536c088dcbe9fc5d"}},{"code_sha256_prefix":"46eea92fab94e361","entry":"load_json","repo":"GuanlinLee/ART","repo_kind":"official","path":"parse_comprehensive_results.py","file_url":"https://github.com/GuanlinLee/ART/blob/HEAD/parse_comprehensive_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46eea92fab94e361"}},{"code_sha256_prefix":"2aef95d6fafac83d","entry":"parse_img_detector_results","repo":"GuanlinLee/ART","repo_kind":"official","path":"parse_comprehensive_results.py","file_url":"https://github.com/GuanlinLee/ART/blob/HEAD/parse_comprehensive_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2aef95d6fafac83d"}},{"code_sha256_prefix":"8effea0d46b99125","entry":"select_a_prompt","repo":"guanlinlee/art","repo_kind":"official","path":"craft_vlm_dataset.py","file_url":"https://github.com/guanlinlee/art/blob/HEAD/craft_vlm_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8effea0d46b99125"}},{"code_sha256_prefix":"98e2d79d448913e2","entry":"select_a_prompt","repo":"GuanlinLee/ART","repo_kind":"official","path":"craft_llm_dataset.py","file_url":"https://github.com/GuanlinLee/ART/blob/HEAD/craft_llm_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98e2d79d448913e2"}},{"code_sha256_prefix":"c3b47216606831e2","entry":"build_default_prompt","repo":"GuanlinLee/ART","repo_kind":"official","path":"prompt_utils_for_llamaguard.py","file_url":"https://github.com/GuanlinLee/ART/blob/HEAD/prompt_utils_for_llamaguard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3b47216606831e2"}},{"code_sha256_prefix":"8ff8b72148aa79ff","entry":"build_llama","repo":"GuanlinLee/ART","repo_kind":"official","path":"run_agents.py","file_url":"https://github.com/GuanlinLee/ART/blob/HEAD/run_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ff8b72148aa79ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}