{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/low-redundant-optimization-for-large-language","title":"Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment","arxiv_id":"2406.12606","date":"2024-06-18","proceeding":null,"authors":["Zhipeng Chen","Kun Zhou","Wayne Xin Zhao","Jingyuan Wang","Ji-Rong Wen"],"abstract":"Large language models (LLMs) are still struggling in aligning with human preference in complex tasks and scenarios. They are prone to overfit into the unexpected patterns or superficial styles in the training data. We conduct an empirical study that only selects the top-10\\% most updated parameters in LLMs for alignment training, and see improvements in the convergence process and final performance. It indicates the existence of redundant neurons in LLMs for alignment training. To reduce its influence, we propose a low-redundant alignment method named \\textbf{ALLO}, focusing on optimizing the most related neurons with the most useful supervised signals. Concretely, we first identify the neurons that are related to the human preference data by a gradient-based strategy, then identify the alignment-related key tokens by reward models for computing loss. Besides, we also decompose the alignment process into the forgetting and learning stages, where we first forget the tokens with unaligned knowledge and then learn aligned knowledge, by updating different ratios of neurons, respectively. Experimental results on 10 datasets have shown the effectiveness of ALLO. Our code and data are available at \\url{https://github.com/RUCAIBox/ALLO}.","url_abs":"https://arxiv.org/abs/2406.12606v2","url_pdf":"https://arxiv.org/pdf/2406.12606v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"low-redundant-optimization-for-large-language","repo_url":"https://github.com/rucaibox/allo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.12606","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12606"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rucaibox/allo","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/RUCAIBox/ALLO","reach":{"status":"ok"}}],"summary":{"ran":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"6242882208f46fc6","entry":"get_reward_prompt","repo":"RUCAIBox/ALLO","repo_kind":"official","path":"preprocess/gen_forgetting_data/gen_tlr.py","file_url":"https://github.com/RUCAIBox/ALLO/blob/HEAD/preprocess/gen_forgetting_data/gen_tlr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6242882208f46fc6"}},{"code_sha256_prefix":"91ce4c9d4d225664","entry":"make_supervised_data_module","repo":"RUCAIBox/ALLO","repo_kind":"official","path":"train/train_forgetting.py","file_url":"https://github.com/RUCAIBox/ALLO/blob/HEAD/train/train_forgetting.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91ce4c9d4d225664"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}