{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-role-of-attention-heads-in-large","title":"On the Role of Attention Heads in Large Language Model Safety","arxiv_id":"2410.13708","date":"2024-10-17","proceeding":null,"authors":["Zhenhong Zhou","Haiyang Yu","Xinghua Zhang","Rongwu Xu","Fei Huang","Kun Wang","Yang Liu","Junfeng Fang","Yongbin Li"],"abstract":"Large language models (LLMs) achieve state-of-the-art performance on multiple language tasks, yet their safety guardrails can be circumvented, leading to harmful generations. In light of this, recent research on safety mechanisms has emerged, revealing that when safety representations or component are suppressed, the safety capability of LLMs are compromised. However, existing research tends to overlook the safety impact of multi-head attention mechanisms, despite their crucial role in various model functionalities. Hence, in this paper, we aim to explore the connection between standard attention mechanisms and safety capability to fill this gap in the safety-related mechanistic interpretability. We propose a novel metric which tailored for multi-head attention, the Safety Head ImPortant Score (Ships), to assess the individual heads' contributions to model safety. Based on this, we generalize Ships to the dataset level and further introduce the Safety Attention Head AttRibution Algorithm (Sahara) to attribute the critical safety attention heads inside the model. Our findings show that the special attention head has a significant impact on safety. Ablating a single safety head allows aligned model (e.g., Llama-2-7b-chat) to respond to 16 times more harmful queries, while only modifying 0.006% of the parameters, in contrast to the ~ 5% modification required in previous studies. More importantly, we demonstrate that attention heads primarily function as feature extractors for safety and models fine-tuned from the same base model exhibit overlapping safety heads through comprehensive experiments. Together, our attribution approach and findings provide a novel perspective for unpacking the black box of safety mechanisms within large models.","url_abs":"https://arxiv.org/abs/2410.13708v2","url_pdf":"https://arxiv.org/pdf/2410.13708v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-role-of-attention-heads-in-large","repo_url":"https://github.com/ydyjya/safetyheadattribution","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.13708","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.13708"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ydyjya/SafetyHeadAttribution","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ydyjya/safetyheadattribution","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":6,"ran":8,"ran_honours":1,"unverified":8},"by_repo_kind":{"official":{"samples":24,"ran":16,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":24,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"ydyjya/SafetyHeadAttribution","repo_kind":"official","path":"lib/utils/custommodel.py","file_url":"https://github.com/ydyjya/SafetyHeadAttribution/blob/HEAD/lib/utils/custommodel.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"583539efd6fd01fb","entry":"apply_rotary_pos_emb","repo":"ydyjya/SafetyHeadAttribution","repo_kind":"official","path":"lib/utils/custommodel.py","file_url":"https://github.com/ydyjya/SafetyHeadAttribution/blob/HEAD/lib/utils/custommodel.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"583539efd6fd01fb"}},{"code_sha256_prefix":"0f1433d59a9e954b","entry":"compute_subspace_similarity","repo":"ydyjya/SafetyHeadAttribution","repo_kind":"official","path":"lib/Sahara/svd.py","file_url":"https://github.com/ydyjya/SafetyHeadAttribution/blob/HEAD/lib/Sahara/svd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f1433d59a9e954b"}},{"code_sha256_prefix":"313de3d0e5c72b22","entry":"get_last_hidden_states","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/Sahara/attribution.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/Sahara/attribution.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"313de3d0e5c72b22"}},{"code_sha256_prefix":"fccfacd5ef8bfee9","entry":"get_tokenizer","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/SHIPS/get_ships.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/SHIPS/get_ships.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fccfacd5ef8bfee9"}},{"code_sha256_prefix":"2c199cf061f618c1","entry":"kl_divergence","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/SHIPS/get_ships.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/SHIPS/get_ships.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c199cf061f618c1"}},{"code_sha256_prefix":"6921907267b978d9","entry":"load_batch_generations","repo":"ydyjya/SafetyHeadAttribution","repo_kind":"official","path":"lib/Safety_discriminator/Discriminator.py","file_url":"https://github.com/ydyjya/SafetyHeadAttribution/blob/HEAD/lib/Safety_discriminator/Discriminator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6921907267b978d9"}},{"code_sha256_prefix":"9ed59a42c3d32d8d","entry":"load_conv","repo":"ydyjya/SafetyHeadAttribution","repo_kind":"official","path":"lib/utils/load_conv.py","file_url":"https://github.com/ydyjya/SafetyHeadAttribution/blob/HEAD/lib/utils/load_conv.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ed59a42c3d32d8d"}},{"code_sha256_prefix":"2b9bd829f84c86a0","entry":"load_from_search_cfg","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/Sahara/attribution.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/Sahara/attribution.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b9bd829f84c86a0"}},{"code_sha256_prefix":"4c67b494ef77df17","entry":"normalize_columns","repo":"ydyjya/SafetyHeadAttribution","repo_kind":"official","path":"lib/Sahara/svd.py","file_url":"https://github.com/ydyjya/SafetyHeadAttribution/blob/HEAD/lib/Sahara/svd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c67b494ef77df17"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"ydyjya/SafetyHeadAttribution","repo_kind":"official","path":"lib/utils/custommodel.py","file_url":"https://github.com/ydyjya/SafetyHeadAttribution/blob/HEAD/lib/utils/custommodel.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"50efcc04b774528b","entry":"run_generation","repo":"ydyjya/SafetyHeadAttribution","repo_kind":"official","path":"lib/utils/batch_inference.py","file_url":"https://github.com/ydyjya/SafetyHeadAttribution/blob/HEAD/lib/utils/batch_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50efcc04b774528b"}},{"code_sha256_prefix":"c11108ae20c7b667","entry":"ships_res_stats","repo":"ydyjya/SafetyHeadAttribution","repo_kind":"official","path":"lib/SHIPS/ships_utils.py","file_url":"https://github.com/ydyjya/SafetyHeadAttribution/blob/HEAD/lib/SHIPS/ships_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c11108ae20c7b667"}},{"code_sha256_prefix":"e222f3baa86eced9","entry":"sort_ships_dict","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/SHIPS/get_ships.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/SHIPS/get_ships.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e222f3baa86eced9"}},{"code_sha256_prefix":"0f7c423b7f272901","entry":"svd_decomposition","repo":"ydyjya/SafetyHeadAttribution","repo_kind":"official","path":"lib/Sahara/svd.py","file_url":"https://github.com/ydyjya/SafetyHeadAttribution/blob/HEAD/lib/Sahara/svd.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f7c423b7f272901"}},{"code_sha256_prefix":"3a2320c99110f0df","entry":"update_mask_cfg","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/Sahara/attribution.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/Sahara/attribution.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3a2320c99110f0df"}},{"code_sha256_prefix":"046f69bea319c560","entry":"CustomLlamaAttention","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/SHIPS/get_ships.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/SHIPS/get_ships.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"046f69bea319c560"}},{"code_sha256_prefix":"fc78267c5d6f1748","entry":"CustomLlamaDecoderLayer","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/SHIPS/get_ships.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/SHIPS/get_ships.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc78267c5d6f1748"}},{"code_sha256_prefix":"f7277da16bd23b13","entry":"CustomLlamaModel","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/SHIPS/get_ships.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/SHIPS/get_ships.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7277da16bd23b13"}},{"code_sha256_prefix":"e8b5246184992c83","entry":"CustomLlamaModelForCausalLM","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/SHIPS/get_ships.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/SHIPS/get_ships.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8b5246184992c83"}},{"code_sha256_prefix":"94bba6aa672d4a96","entry":"CustomMistralModelForCausalLM","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/SHIPS/get_ships.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/SHIPS/get_ships.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94bba6aa672d4a96"}},{"code_sha256_prefix":"d2287a55acb3b732","entry":"SHIPS","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/SHIPS/get_ships.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/SHIPS/get_ships.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2287a55acb3b732"}},{"code_sha256_prefix":"9cddf5ae9689b5a0","entry":"get_model","repo":"ydyjya/safetyheadattribution","repo_kind":"official","path":"lib/SHIPS/get_ships.py","file_url":"https://github.com/ydyjya/safetyheadattribution/blob/HEAD/lib/SHIPS/get_ships.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9cddf5ae9689b5a0"}},{"code_sha256_prefix":"92f77d2126ff0672","entry":"load_jsonl2text_list","repo":"ydyjya/SafetyHeadAttribution","repo_kind":"official","path":"lib/Safety_discriminator/Discriminator.py","file_url":"https://github.com/ydyjya/SafetyHeadAttribution/blob/HEAD/lib/Safety_discriminator/Discriminator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92f77d2126ff0672"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}