{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2512-11391","title":"Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization","arxiv_id":"2512.11391","date":"2025-12-12","proceeding":"ICLR","authors":["Yifan Niu","Han Xiao","Dongyi Liu","Nuo Chen","Jia Li"],"abstract":"As Large Language Models (LLMs) are increasingly deployed in real-world applications, it is important to ensure their behaviors align with human values, societal norms, and ethical principles. However, safety alignment under Reinforcement Learning (RL) often suffers from forgetting learned general abilities, which is also known as the alignment tax. To address this issue, we introduce Null-Space constrained Policy Optimization (NSPO), a novel RL framework for LLM safety alignment while preserving their core abilities. The safety policy gradients are geometrically projected into the null space of general tasks, thereby mitigating the safety alignment tax. In addition, we theoretically prove that NSPO preserves the model's original core capabilities, while still guaranteeing a descent direction for effective safety alignment. Extensive experiments demonstrate that NSPO outperforms existing methods by a large margin, achieving state-of-the-art safety performance without sacrificing accuracy on general tasks, including math, code, and instruction-following tasks. Notably, NSPO is data-efficient and only requires 40% of public human-annotated safety data from PKU-SafeRLHF to achieve promising safety performance, without a large amount of mixed general tasks data in existing alignment methods.","url_abs":"https://arxiv.org/abs/2512.11391","url_pdf":"https://arxiv.org/pdf/2512.11391","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2512.11391","atlas_url":"https://app.syntology.ai/?focus=2512.11391","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2512.11391"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ivanniu/NSPO","reach":{"status":"ok"}}],"summary":{"ran":3,"ran_draft_wrong":3,"unverified":6},"by_repo_kind":{"found_in_text":{"samples":12,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"f220595adfc5e83c","entry":"compute_ce_dpo_loss_rm","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"verl/recipe/prime/prime_core_algos.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/verl/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f220595adfc5e83c"}},{"code_sha256_prefix":"4d91e4383716e42c","entry":"compute_detach_dpo_loss_rm","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"verl/recipe/prime/prime_core_algos.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/verl/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d91e4383716e42c"}},{"code_sha256_prefix":"f19825983a523003","entry":"compute_online_dpo_loss","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"verl/recipe/spin/core_algos.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/verl/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f19825983a523003"}},{"code_sha256_prefix":"41b6e81dc3ecba79","entry":"compute_onlinedpo_pref","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"verl/recipe/spin/core_algos.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/verl/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"41b6e81dc3ecba79"}},{"code_sha256_prefix":"757d109d90121142","entry":"get_kl_controller","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"verl/recipe/spin/core_algos.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/verl/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"757d109d90121142"}},{"code_sha256_prefix":"c30b2a96103abd2c","entry":"reward_func","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"verl/recipe/r1/reward_score.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/verl/recipe/r1/reward_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c30b2a96103abd2c"}},{"code_sha256_prefix":"9ba031d07c0634c0","entry":"avg","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"evaluation/eval_sorrybench.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/evaluation/eval_sorrybench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ba031d07c0634c0"}},{"code_sha256_prefix":"1b1cdeb2c61f601c","entry":"extract_res","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"evaluation/evaluation.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/evaluation/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1b1cdeb2c61f601c"}},{"code_sha256_prefix":"2d708a10ebb6bdd8","entry":"get_PKU","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"script/standard_dpo.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/script/standard_dpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d708a10ebb6bdd8"}},{"code_sha256_prefix":"6d129731f3e8f3a7","entry":"get_scores","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"script/safe_reward.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/script/safe_reward.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d129731f3e8f3a7"}},{"code_sha256_prefix":"9a8ba8234f2d3f9a","entry":"return_back_input","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"evaluation/evaluation.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/evaluation/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a8ba8234f2d3f9a"}},{"code_sha256_prefix":"3beb94ccc55eeeb1","entry":"return_back_prompt_template","repo":"ivanniu/NSPO","repo_kind":"found_in_text","path":"evaluation/evaluation.py","file_url":"https://github.com/ivanniu/NSPO/blob/HEAD/evaluation/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3beb94ccc55eeeb1"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}