{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fine-tuning-linear-layers-only-is-a-simple","title":"Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmetic","arxiv_id":"2407.07089","date":"2024-07-09","proceeding":null,"authors":["Ruochen Jin","BoJian Hou","Jiancong Xiao","Weijie Su","Li Shen"],"abstract":"In recent years, task arithmetic has garnered increasing attention. This approach edits pre-trained models directly in weight space by combining the fine-tuned weights of various tasks into a unified model. Its efficiency and cost-effectiveness stem from its training-free combination, contrasting with traditional methods that require model training on large datasets for multiple tasks. However, applying such a unified model to individual tasks can lead to interference from other tasks (lack of weight disentanglement). To address this issue, Neural Tangent Kernel (NTK) linearization has been employed to leverage a \"kernel behavior\", facilitating weight disentanglement and mitigating adverse effects from unrelated tasks. Despite its benefits, NTK linearization presents drawbacks, including doubled training costs, as well as reduced performance of individual models. To tackle this problem, we propose a simple yet effective and efficient method that is to finetune the attention modules only in the Transformer. Our study reveals that the attention modules exhibit kernel behavior, and fine-tuning the attention modules only significantly improves weight disentanglement. To further understand how our method improves the weight disentanglement of task arithmetic, we present a comprehensive study of task arithmetic by differentiating the role of the representation module and task-specific module. In particular, we find that the representation module plays an important role in improving weight disentanglement whereas the task-specific modules such as the classification heads can degenerate the weight disentanglement performance. (The code is available at https://github.com/kyrie-23/task_arithmetic_tangent)","url_abs":"https://arxiv.org/abs/2407.07089v2","url_pdf":"https://arxiv.org/pdf/2407.07089v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fine-tuning-linear-layers-only-is-a-simple","repo_url":"https://github.com/kyrie-23/task_arithmetic_tangent","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"fine-tuning-linear-layers-only-is-a-simple","repo_url":"https://github.com/kyrie-23/linear_task_arithmetic","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"task-arithmetic","task_name":"Task Arithmetic"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.07089","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.07089"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kyrie-23/linear_task_arithmetic","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kyrie-23/task_arithmetic_tangent","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3053d2bdc3ea5c89","entry":"create_log_dir","repo":"kyrie-23/task_arithmetic_tangent","repo_kind":"official","path":"src/main_task_wise_adamerging.py","file_url":"https://github.com/kyrie-23/task_arithmetic_tangent/blob/HEAD/src/main_task_wise_adamerging.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3053d2bdc3ea5c89"}},{"code_sha256_prefix":"8adbe30621fbeb18","entry":"make_functional","repo":"kyrie-23/task_arithmetic_tangent","repo_kind":"official","path":"src/main_task_wise_adamerging.py","file_url":"https://github.com/kyrie-23/task_arithmetic_tangent/blob/HEAD/src/main_task_wise_adamerging.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8adbe30621fbeb18"}},{"code_sha256_prefix":"c2df94ca3de67eb0","entry":"softmax_entropy","repo":"kyrie-23/task_arithmetic_tangent","repo_kind":"official","path":"src/main_task_wise_adamerging.py","file_url":"https://github.com/kyrie-23/task_arithmetic_tangent/blob/HEAD/src/main_task_wise_adamerging.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2df94ca3de67eb0"}},{"code_sha256_prefix":"74b359c7b8574568","entry":"optimize_disentanglement","repo":"kyrie-23/linear_task_arithmetic","repo_kind":"official","path":"src/optimize.py","file_url":"https://github.com/kyrie-23/linear_task_arithmetic/blob/HEAD/src/optimize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74b359c7b8574568"}},{"code_sha256_prefix":"0b6df3371834b271","entry":"parse_arguments","repo":"kyrie-23/linear_task_arithmetic","repo_kind":"official","path":"src/optimize.py","file_url":"https://github.com/kyrie-23/linear_task_arithmetic/blob/HEAD/src/optimize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b6df3371834b271"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}