{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adarank-adaptive-rank-pruning-for-enhanced","title":"AdaRank: Adaptive Rank Pruning for Enhanced Model Merging","arxiv_id":"2503.22178","date":"2025-03-28","proceeding":null,"authors":["Chanhyuk Lee","Jiho Choi","Chanryeol Lee","Donggyun Kim","Seunghoon Hong"],"abstract":"Model merging has emerged as a promising approach for unifying independently fine-tuned models into an integrated framework, significantly enhancing computational efficiency in multi-task learning. Recently, several SVD-based techniques have been introduced to exploit low-rank structures for enhanced merging, but their reliance on such manually designed rank selection often leads to cross-task interference and suboptimal performance. In this paper, we propose AdaRank, a novel model merging framework that adaptively selects the most beneficial singular directions of task vectors to merge multiple models. We empirically show that the dominant singular components of task vectors can cause critical interference with other tasks, and that naive truncation across tasks and layers degrades performance. In contrast, AdaRank dynamically prunes the singular components that cause interference and offers an optimal amount of information to each task vector by learning to prune ranks during test-time via entropy minimization. Our analysis demonstrates that such method mitigates detrimental overlaps among tasks, while empirical results show that AdaRank consistently achieves state-of-the-art performance with various backbones and number of tasks, reducing the performance gap between fine-tuned models to nearly 1%.","url_abs":"https://arxiv.org/abs/2503.22178v1","url_pdf":"https://arxiv.org/pdf/2503.22178v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adarank-adaptive-rank-pruning-for-enhanced","repo_url":"https://github.com/david3684/adarank","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.22178","atlas_url":"https://app.syntology.ai/?focus=2503.22178","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.22178"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/david3684/adarank","reach":null}],"summary":{"ran":1,"ran_draft_wrong":2,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"d24346902b4fa7e1","entry":"TaskVector","repo":"david3684/adarank","repo_kind":"official","path":"lm/model_merging_methods/adarank_gpt2.py","file_url":"https://github.com/david3684/adarank/blob/HEAD/lm/model_merging_methods/adarank_gpt2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d24346902b4fa7e1"}},{"code_sha256_prefix":"64094ca096eebd1c","entry":"get_exclude_regex","repo":"david3684/adarank","repo_kind":"official","path":"lm/model_merging_methods/adarank_gpt2.py","file_url":"https://github.com/david3684/adarank/blob/HEAD/lm/model_merging_methods/adarank_gpt2.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64094ca096eebd1c"}},{"code_sha256_prefix":"840199cfb76e2913","entry":"get_param_names_to_merge","repo":"david3684/adarank","repo_kind":"official","path":"lm/model_merging_methods/adarank_gpt2.py","file_url":"https://github.com/david3684/adarank/blob/HEAD/lm/model_merging_methods/adarank_gpt2.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"840199cfb76e2913"}},{"code_sha256_prefix":"25c594e39ce9e318","entry":"is_mat_params","repo":"david3684/adarank","repo_kind":"official","path":"lm/model_merging_methods/adarank_gpt2.py","file_url":"https://github.com/david3684/adarank/blob/HEAD/lm/model_merging_methods/adarank_gpt2.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"25c594e39ce9e318"}},{"code_sha256_prefix":"023a17baf96e0b3d","entry":"AdaRankModule","repo":"david3684/adarank","repo_kind":"official","path":"lm/model_merging_methods/adarank_gpt2.py","file_url":"https://github.com/david3684/adarank/blob/HEAD/lm/model_merging_methods/adarank_gpt2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"023a17baf96e0b3d"}},{"code_sha256_prefix":"795a4a836a6ee6b5","entry":"truncated_svd","repo":"david3684/adarank","repo_kind":"official","path":"lm/model_merging_methods/adarank_gpt2.py","file_url":"https://github.com/david3684/adarank/blob/HEAD/lm/model_merging_methods/adarank_gpt2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"795a4a836a6ee6b5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}