{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llm-safety-alignment-is-divergence-estimation","title":"LLM Safety Alignment is Divergence Estimation in Disguise","arxiv_id":"2502.00657","date":"2025-02-02","proceeding":null,"authors":["Rajdeep Haldar","Ziyi Wang","Qifan Song","Guang Lin","Yue Xing"],"abstract":"We present a theoretical framework showing that popular LLM alignment methods, including RLHF and its variants, can be understood as divergence estimators between aligned (safe or preferred) and unaligned (harmful or less preferred) distributions. This perspective explains the emergence of separation in the latent space between safe and harmful prompts after alignment. As an application of our general divergence framework, we propose KLDO, a novel KL divergence-based alignment method, and empirically validate its effectiveness. We further show that using compliance-refusal datasets, rather than standard preference-based datasets, leads to stronger separation and improved safety alignment. Finally, to quantify the separation effect, we propose a distance-based metric in the prompt representation space, which also acts as a statistically significant indicator for model safety.","url_abs":"https://arxiv.org/abs/2502.00657v2","url_pdf":"https://arxiv.org/pdf/2502.00657v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llm-safety-alignment-is-divergence-estimation","repo_url":"https://github.com/rhaldarpurdue/kldo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"safety-alignment","task_name":"Safety Alignment"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.00657","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.00657"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rhaldarpurdue/kldo","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":16},"by_repo_kind":{"official":{"samples":17,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"762172a898132a82","entry":"generate","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"generate.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/generate.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"762172a898132a82"}},{"code_sha256_prefix":"a3e5e980b0266d7e","entry":"apply_chat_template","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a3e5e980b0266d7e"}},{"code_sha256_prefix":"55f147c931c65d11","entry":"bhattacharyya_coeff","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"bhatta_dist.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/bhatta_dist.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55f147c931c65d11"}},{"code_sha256_prefix":"33621a50f5a245fb","entry":"bhattacharyya_distance_multivariate","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"bhatta_dist.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/bhatta_dist.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"33621a50f5a245fb"}},{"code_sha256_prefix":"740872830c85598c","entry":"generate_accepted","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"dataset_generation/generate.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/dataset_generation/generate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"740872830c85598c"}},{"code_sha256_prefix":"f586d1feb30e8b1f","entry":"generate_rejected","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"dataset_generation/generate.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/dataset_generation/generate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f586d1feb30e8b1f"}},{"code_sha256_prefix":"73d2eea3309d497e","entry":"get_hidden_states","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73d2eea3309d497e"}},{"code_sha256_prefix":"9558b03087afac86","entry":"get_not_allowed_tokens","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"eval_utils.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9558b03087afac86"}},{"code_sha256_prefix":"962f72a6b771258f","entry":"get_response","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"dataset_generation/compare.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/dataset_generation/compare.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"962f72a6b771258f"}},{"code_sha256_prefix":"4cc483f4696c2521","entry":"load_dataset","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"eval_utils.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4cc483f4696c2521"}},{"code_sha256_prefix":"63fd436cd288e12f","entry":"load_model_and_tokenizer","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"eval_utils.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/eval_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"63fd436cd288e12f"}},{"code_sha256_prefix":"d690eaa700e23094","entry":"load_model_and_tokenizer","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"dataset_generation/compare.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/dataset_generation/compare.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d690eaa700e23094"}},{"code_sha256_prefix":"b95843089da1efe9","entry":"load_model_and_tokenizer","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"dataset_generation/generate.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/dataset_generation/generate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b95843089da1efe9"}},{"code_sha256_prefix":"36bb4333595d757f","entry":"pca_reduce_dimensions","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"36bb4333595d757f"}},{"code_sha256_prefix":"2d598ab5942a0271","entry":"prepend_sys_prompt","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"generate.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/generate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d598ab5942a0271"}},{"code_sha256_prefix":"da8ca4544aefb294","entry":"process_soft_prompt_as_word_embedding","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"generate.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/generate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da8ca4544aefb294"}},{"code_sha256_prefix":"8257f273d387a13e","entry":"response_list","repo":"rhaldarpurdue/kldo","repo_kind":"official","path":"dataset_generation/compare.py","file_url":"https://github.com/rhaldarpurdue/kldo/blob/HEAD/dataset_generation/compare.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8257f273d387a13e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}