{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/red-teaming-language-models-with-language","title":"Red Teaming Language Models with Language Models","arxiv_id":"2202.03286","date":"2022-02-07","proceeding":null,"authors":["Ethan Perez","Saffron Huang","Francis Song","Trevor Cai","Roman Ring","John Aslanides","Amelia Glaese","Nat McAleese","Geoffrey Irving"],"abstract":"Language Models (LMs) often cannot be deployed because of their potential to harm users in hard-to-predict ways. Prior work identifies harmful behaviors before deployment by using human annotators to hand-write test cases. However, human annotation is expensive, limiting the number and diversity of test cases. In this work, we automatically find cases where a target LM behaves in a harmful way, by generating test cases (\"red teaming\") using another LM. We evaluate the target LM's replies to generated test questions using a classifier trained to detect offensive content, uncovering tens of thousands of offensive replies in a 280B parameter LM chatbot. We explore several methods, from zero-shot generation to reinforcement learning, for generating test cases with varying levels of diversity and difficulty. Furthermore, we use prompt engineering to control LM-generated test cases to uncover a variety of other harms, automatically finding groups of people that the chatbot discusses in offensive ways, personal and hospital phone numbers generated as the chatbot's own contact info, leakage of private training data in generated text, and harms that occur over the course of a conversation. Overall, LM-based red teaming is one promising tool (among many needed) for finding and fixing diverse, undesirable LM behaviors before impacting users.","url_abs":"https://arxiv.org/abs/2202.03286v1","url_pdf":"https://arxiv.org/pdf/2202.03286v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"red-teaming-language-models-with-language","repo_url":"https://github.com/shreyansh26/Red-Teaming-Language-Models-with-Language-Models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"chatbot","task_name":"Chatbot"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"},{"task_slug":"red-teaming","task_name":"Red Teaming"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.03286","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.03286"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shreyansh26/Red-Teaming-Language-Models-with-Language-Models","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"4d854d88e070847f","entry":"swor_gumbel","repo":"shreyansh26/Red-Teaming-Language-Models-with-Language-Models","repo_kind":"listed","path":"generate_questions_few_shot.py","file_url":"https://github.com/shreyansh26/Red-Teaming-Language-Models-with-Language-Models/blob/HEAD/generate_questions_few_shot.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d854d88e070847f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}