{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2509-00544","title":"When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignment","arxiv_id":"2509.00544","date":"2025-08-30","proceeding":"ICLR","authors":["Hanqi Yan","Hainiu Xu","Siya Qi","Shu Yang","Yulan He"],"abstract":"With the growing accessibility and wide adoption of large language models, concerns about their safety and alignment with human values have become paramount. In this paper, we identify a concerning phenomenon: Reasoning-Induced Misalignment (RIM), in which misalignment emerges when reasoning capabilities strengthened-particularly when specific types of reasoning patterns are introduced during inference or training. Beyond reporting this vulnerability, we provide the first mechanistic account of its origins. Through representation analysis, we discover that specific attention heads facilitate refusal by reducing their attention to CoT tokens, a mechanism that modulates the model's rationalization process during inference. During training, we find significantly higher activation entanglement between reasoning and safety in safety-critical neurons than in control neurons, particularly after fine-tuning with those identified reasoning patterns. This entanglement strongly correlates with catastrophic forgetting, providing a neuron-level explanation for RIM.","url_abs":"https://arxiv.org/abs/2509.00544","url_pdf":"https://arxiv.org/pdf/2509.00544","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2509.00544","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2509.00544"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/seacowx/When-Thinking-Backfires","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":4,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"1024978ec8aedf27","entry":"get_activation_shrinkage","repo":"seacowx/When-Thinking-Backfires","repo_kind":"found_in_text","path":"RAS/compute_entanglement.py","file_url":"https://github.com/seacowx/When-Thinking-Backfires/blob/HEAD/RAS/compute_entanglement.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1024978ec8aedf27"}},{"code_sha256_prefix":"d1f0767cd7b7ac40","entry":"get_entangled_neuron_idx_list","repo":"seacowx/When-Thinking-Backfires","repo_kind":"found_in_text","path":"RAS/compute_entanglement.py","file_url":"https://github.com/seacowx/When-Thinking-Backfires/blob/HEAD/RAS/compute_entanglement.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1f0767cd7b7ac40"}},{"code_sha256_prefix":"b25ab8aa4af3b611","entry":"load_activations","repo":"seacowx/When-Thinking-Backfires","repo_kind":"found_in_text","path":"RAS/compute_entanglement.py","file_url":"https://github.com/seacowx/When-Thinking-Backfires/blob/HEAD/RAS/compute_entanglement.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b25ab8aa4af3b611"}},{"code_sha256_prefix":"8b87263c36d7f4be","entry":"organize_lora_response","repo":"seacowx/When-Thinking-Backfires","repo_kind":"found_in_text","path":"inference/eval_hexphi.py","file_url":"https://github.com/seacowx/When-Thinking-Backfires/blob/HEAD/inference/eval_hexphi.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b87263c36d7f4be"}},{"code_sha256_prefix":"869bf93ad159dadb","entry":"organize_ordinary_response","repo":"seacowx/When-Thinking-Backfires","repo_kind":"found_in_text","path":"inference/eval_hexphi.py","file_url":"https://github.com/seacowx/When-Thinking-Backfires/blob/HEAD/inference/eval_hexphi.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"869bf93ad159dadb"}},{"code_sha256_prefix":"cfa1daa01b202035","entry":"prepare_data","repo":"seacowx/When-Thinking-Backfires","repo_kind":"found_in_text","path":"inference/eval_hexphi.py","file_url":"https://github.com/seacowx/When-Thinking-Backfires/blob/HEAD/inference/eval_hexphi.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfa1daa01b202035"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_api"},"syntology_extracted_results":null}