{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/refusal-in-language-models-is-mediated-by-a","title":"Refusal in Language Models Is Mediated by a Single Direction","arxiv_id":"2406.11717","date":"2024-06-17","proceeding":null,"authors":["Andy Arditi","Oscar Obeso","Aaquib Syed","Daniel Paleka","Nina Panickssery","Wes Gurnee","Neel Nanda"],"abstract":"Conversational large language models are fine-tuned for both instruction-following and safety, resulting in models that obey benign requests but refuse harmful ones. While this refusal behavior is widespread across chat models, its underlying mechanisms remain poorly understood. In this work, we show that refusal is mediated by a one-dimensional subspace, across 13 popular open-source chat models up to 72B parameters in size. Specifically, for each model, we find a single direction such that erasing this direction from the model's residual stream activations prevents it from refusing harmful instructions, while adding this direction elicits refusal on even harmless instructions. Leveraging this insight, we propose a novel white-box jailbreak method that surgically disables refusal with minimal effect on other capabilities. Finally, we mechanistically analyze how adversarial suffixes suppress propagation of the refusal-mediating direction. Our findings underscore the brittleness of current safety fine-tuning methods. More broadly, our work showcases how an understanding of model internals can be leveraged to develop practical methods for controlling model behavior.","url_abs":"https://arxiv.org/abs/2406.11717v3","url_pdf":"https://arxiv.org/pdf/2406.11717v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"refusal-in-language-models-is-mediated-by-a","repo_url":"https://github.com/andyrdt/refusal_direction","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.11717","atlas_url":"https://app.syntology.ai/?focus=2406.11717","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.11717"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/andyrdt/refusal_direction","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":12,"unverified":3},"by_repo_kind":{"official":{"samples":15,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cfecd6ef02a68616","entry":"format_instruction_gemma_chat","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/model_utils/gemma_model.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/model_utils/gemma_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cfecd6ef02a68616"}},{"code_sha256_prefix":"e433774fbe719204","entry":"format_instruction_llama2_chat","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/model_utils/llama2_model.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/model_utils/llama2_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e433774fbe719204"}},{"code_sha256_prefix":"79d4e41de6f143f3","entry":"format_instruction_llama3_chat","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/model_utils/llama3_model.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/model_utils/llama3_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"79d4e41de6f143f3"}},{"code_sha256_prefix":"91bb65bbcbe02944","entry":"format_instruction_qwen_chat","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/model_utils/qwen_model.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/model_utils/qwen_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"91bb65bbcbe02944"}},{"code_sha256_prefix":"2f0cbbb3c1d6dfcf","entry":"format_instruction_yi_chat","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/model_utils/yi_model.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/model_utils/yi_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f0cbbb3c1d6dfcf"}},{"code_sha256_prefix":"af35abde0810ea46","entry":"refusal_score","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/submodules/select_direction.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/submodules/select_direction.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af35abde0810ea46"}},{"code_sha256_prefix":"5ccac6a33ca73bb5","entry":"substring_matching_judge_fn","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/submodules/evaluate_jailbreak.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/submodules/evaluate_jailbreak.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ccac6a33ca73bb5"}},{"code_sha256_prefix":"57abe23529aab499","entry":"tokenize_instructions_gemma_chat","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/model_utils/gemma_model.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/model_utils/gemma_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57abe23529aab499"}},{"code_sha256_prefix":"00bf117f8b25ddae","entry":"tokenize_instructions_llama2_chat","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/model_utils/llama2_model.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/model_utils/llama2_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"00bf117f8b25ddae"}},{"code_sha256_prefix":"c180a112405739dc","entry":"tokenize_instructions_llama3_chat","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/model_utils/llama3_model.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/model_utils/llama3_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c180a112405739dc"}},{"code_sha256_prefix":"2ea9d470cff77a2b","entry":"tokenize_instructions_qwen_chat","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/model_utils/qwen_model.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/model_utils/qwen_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2ea9d470cff77a2b"}},{"code_sha256_prefix":"e43f427d3fdea028","entry":"tokenize_instructions_yi_chat","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/model_utils/yi_model.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/model_utils/yi_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e43f427d3fdea028"}},{"code_sha256_prefix":"9676feab77ab828b","entry":"llamaguard2_judge_fn","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"pipeline/submodules/evaluate_jailbreak.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/pipeline/submodules/evaluate_jailbreak.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9676feab77ab828b"}},{"code_sha256_prefix":"1e052ccbbf393dbe","entry":"load_dataset","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"dataset/load_dataset.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/dataset/load_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e052ccbbf393dbe"}},{"code_sha256_prefix":"a746c0ec00a96511","entry":"load_dataset_split","repo":"andyrdt/refusal_direction","repo_kind":"official","path":"dataset/load_dataset.py","file_url":"https://github.com/andyrdt/refusal_direction/blob/HEAD/dataset/load_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a746c0ec00a96511"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}