{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/detox-toxic-subspace-projection-for-model","title":"Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity","arxiv_id":"2405.13967","date":"2024-05-22","proceeding":null,"authors":["Rheeya Uppaal","Apratim Dey","Yiting He","Yiqiao Zhong","Junjie Hu"],"abstract":"Recent alignment algorithms such as direct preference optimization (DPO) have been developed to improve the safety of large language models (LLMs) by training these models to match human behaviors exemplified by preference data. However, these methods are both computationally intensive and lacking in controllability and transparency, inhibiting their widespread use. Furthermore, these tuning-based methods require large-scale preference data for training and are susceptible to noisy preference data. In this paper, we introduce a tuning-free alignment alternative, ProFS (Projection Filter for Subspaces), and demonstrate its effectiveness under the use case of toxicity reduction. Grounded on theory from factor analysis, ProFS is a sample-efficient model editing approach that identifies a toxic subspace in the model parameter space and reduces model toxicity by projecting away the detected subspace. The toxic subspace is identified by extracting preference data embeddings from the language model, and removing non-toxic information from these embeddings. We show that ProFS is more sample-efficient than DPO, further showcasing greater robustness to noisy data. Finally, we attempt to connect tuning based alignment with editing, by establishing both theoretical and empirical connections between ProFS and DPO, showing that ProFS can be interpreted as a denoised version of a single DPO step.","url_abs":"https://arxiv.org/abs/2405.13967v4","url_pdf":"https://arxiv.org/pdf/2405.13967v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"detox-toxic-subspace-projection-for-model","repo_url":"https://github.com/uppaal/detox-edit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"detox-toxic-subspace-projection-for-model","repo_url":"https://github.com/ziwei-zheng/nullu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"model-editing","task_name":"Model Editing"}],"methods":[{"method_slug":"dpo","method_name":"DPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.13967","atlas_url":"https://app.syntology.ai/?focus=2405.13967","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.13967"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Uppaal/detox-edit","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ziwei-zheng/nullu","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uppaal/detox-edit","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":6,"unverified":5},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fd8cb94482303177","entry":"get_continuation","repo":"uppaal/detox-edit","repo_kind":"official","path":"evaluation/win_rate.py","file_url":"https://github.com/uppaal/detox-edit/blob/HEAD/evaluation/win_rate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd8cb94482303177"}},{"code_sha256_prefix":"772f368569ca0f5d","entry":"get_num_transformer_layers","repo":"Uppaal/detox-edit","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/Uppaal/detox-edit/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"772f368569ca0f5d"}},{"code_sha256_prefix":"ec4fdca101330482","entry":"load_toxicity_preference","repo":"Uppaal/detox-edit","repo_kind":"official","path":"utils/dataset_utils.py","file_url":"https://github.com/Uppaal/detox-edit/blob/HEAD/utils/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec4fdca101330482"}},{"code_sha256_prefix":"cfb7124a6f05db32","entry":"main","repo":"Uppaal/detox-edit","repo_kind":"official","path":"utils/startup.py","file_url":"https://github.com/Uppaal/detox-edit/blob/HEAD/utils/startup.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfb7124a6f05db32"}},{"code_sha256_prefix":"7bf4ed7f8521c691","entry":"reduce_num_dps","repo":"Uppaal/detox-edit","repo_kind":"official","path":"utils/dataset_utils.py","file_url":"https://github.com/Uppaal/detox-edit/blob/HEAD/utils/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7bf4ed7f8521c691"}},{"code_sha256_prefix":"3dd8a319f62d60ec","entry":"vector_matrix_cosine_similarity","repo":"Uppaal/detox-edit","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/Uppaal/detox-edit/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3dd8a319f62d60ec"}},{"code_sha256_prefix":"aa5ffddfcd573d9a","entry":"vector_vector_cosine_similarity","repo":"Uppaal/detox-edit","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/Uppaal/detox-edit/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5ffddfcd573d9a"}},{"code_sha256_prefix":"6b9b938bdb4d6f5f","entry":"HalluEdit","repo":"ziwei-zheng/nullu","repo_kind":"listed","path":"utils/halluedit.py","file_url":"https://github.com/ziwei-zheng/nullu/blob/HEAD/utils/halluedit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b9b938bdb4d6f5f"}},{"code_sha256_prefix":"e61d0c538220694e","entry":"get_current_gpu_use","repo":"Uppaal/detox-edit","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/Uppaal/detox-edit/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e61d0c538220694e"}},{"code_sha256_prefix":"6387304a152872eb","entry":"get_model_category","repo":"Uppaal/detox-edit","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/Uppaal/detox-edit/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6387304a152872eb"}},{"code_sha256_prefix":"fe59b113554271a7","entry":"llm_judge_eval","repo":"Uppaal/detox-edit","repo_kind":"official","path":"evaluation/win_rate.py","file_url":"https://github.com/Uppaal/detox-edit/blob/HEAD/evaluation/win_rate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe59b113554271a7"}},{"code_sha256_prefix":"78599b638be017da","entry":"load_large_model","repo":"Uppaal/detox-edit","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/Uppaal/detox-edit/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78599b638be017da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}