{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-diverse-attacks-on-large-language","title":"Learning diverse attacks on large language models for robust red-teaming and safety tuning","arxiv_id":"2405.18540","date":"2024-05-28","proceeding":null,"authors":["Seanie Lee","Minsu Kim","Lynn Cherif","David Dobre","Juho Lee","Sung Ju Hwang","Kenji Kawaguchi","Gauthier Gidel","Yoshua Bengio","Nikolay Malkin","Moksh Jain"],"abstract":"Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe and responsible deployment of large language models (LLMs). Developing effective protection against many modes of attack prompts requires discovering diverse attacks. Automated red-teaming typically uses reinforcement learning to fine-tune an attacker language model to generate prompts that elicit undesirable responses from a target LLM, as measured, for example, by an auxiliary toxicity classifier. We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks. As a flexible and probabilistically principled alternative, we propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts. We find that the attacks generated by our method are effective against a wide range of target LLMs, both with and without safety tuning, and transfer well between target LLMs. Finally, we demonstrate that models safety-tuned using a dataset of red-teaming prompts generated by our method are robust to attacks from other RL-based red-teaming approaches.","url_abs":"https://arxiv.org/abs/2405.18540v2","url_pdf":"https://arxiv.org/pdf/2405.18540v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-diverse-attacks-on-large-language","repo_url":"https://github.com/GFNOrg/red-teaming","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"red-teaming","task_name":"Red Teaming"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.18540","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.18540"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/GFNOrg/red-teaming","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9e3fca8886c2c85e","entry":"avg_pooling","repo":"GFNOrg/red-teaming","repo_kind":"official","path":"trainers/gfn_trainer.py","file_url":"https://github.com/GFNOrg/red-teaming/blob/HEAD/trainers/gfn_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e3fca8886c2c85e"}},{"code_sha256_prefix":"0c83713b19aeca88","entry":"batch_cosine_similarity_kernel","repo":"GFNOrg/red-teaming","repo_kind":"official","path":"utils.py","file_url":"https://github.com/GFNOrg/red-teaming/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c83713b19aeca88"}},{"code_sha256_prefix":"9fa453cce5ebc8e9","entry":"check_filename","repo":"GFNOrg/red-teaming","repo_kind":"official","path":"collect_samples.py","file_url":"https://github.com/GFNOrg/red-teaming/blob/HEAD/collect_samples.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fa453cce5ebc8e9"}},{"code_sha256_prefix":"a31a641c4e49e500","entry":"get_dataloader","repo":"GFNOrg/red-teaming","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/GFNOrg/red-teaming/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a31a641c4e49e500"}},{"code_sha256_prefix":"38c281ea8977e9ae","entry":"get_decay_parameter_names","repo":"GFNOrg/red-teaming","repo_kind":"official","path":"utils.py","file_url":"https://github.com/GFNOrg/red-teaming/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"38c281ea8977e9ae"}},{"code_sha256_prefix":"214442c552c26bab","entry":"get_parameter_names","repo":"GFNOrg/red-teaming","repo_kind":"official","path":"utils.py","file_url":"https://github.com/GFNOrg/red-teaming/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"214442c552c26bab"}},{"code_sha256_prefix":"5e006375b39e3def","entry":"make_chat_prompt","repo":"gfnorg/red-teaming","repo_kind":"official","path":"safety_dataset/create_safety_response.py","file_url":"https://github.com/gfnorg/red-teaming/blob/HEAD/safety_dataset/create_safety_response.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e006375b39e3def"}},{"code_sha256_prefix":"1c61f4ae1d677417","entry":"generate_and_return_z_logprob","repo":"GFNOrg/red-teaming","repo_kind":"official","path":"trainers/gfn_trainer.py","file_url":"https://github.com/GFNOrg/red-teaming/blob/HEAD/trainers/gfn_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c61f4ae1d677417"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}