{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/della-merging-reducing-interference-in-model","title":"DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling","arxiv_id":"2406.11617","date":"2024-06-17","proceeding":null,"authors":["Pala Tej Deep","Rishabh Bhardwaj","Soujanya Poria"],"abstract":"With the proliferation of domain-specific models, model merging has emerged as a set of techniques that combine the capabilities of multiple models into one that can multitask without the cost of additional training. In this paper, we propose a new model merging technique, Drop and rEscaLe via sampLing with mAgnitude (DELLA-Merging), that employs a novel pruning technique, MAGPRUNE, which shows significant advantages over DARE and TIES. MAGPRUNE first ranks the parameters in order of their magnitude and assigns higher dropout probabilities (p) to parameters with lower ranks corresponding to lower magnitudes. To approximate the original embeddings, MAGPRUNE employs a rescaling operation on the parameters that survive the random dropping by 1/(1 - p). On three different expert models considered for merging (LM, Math, Code) and corresponding benchmark datasets (AlpacaEval, GSM8K, MBPP), DELLA shows an average improvement of 2.4 points over baseline methods employing delta parameter pruning (an improvement of 3.6 points over TIES, 1.2 points over DARE), and 11.1 points over the no-pruning baseline (TA). We release the source code at: https://github.com/declare-lab/della.","url_abs":"https://arxiv.org/abs/2406.11617v1","url_pdf":"https://arxiv.org/pdf/2406.11617v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"della-merging-reducing-interference-in-model","repo_url":"https://github.com/declare-lab/della","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"math","task_name":"Math"},{"task_slug":null,"task_name":"mbpp"}],"methods":[{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2406.11617","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.11617"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/declare-lab/della","reach":{"status":"ok"}}],"summary":{"ran":9,"unverified":6},"by_repo_kind":{"official":{"samples":15,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"5b1af7ea3edb8c0c","entry":"dtype_from_name","repo":"declare-lab/della","repo_kind":"official","path":"mergekit/mergekit/common.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/mergekit/mergekit/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b1af7ea3edb8c0c"}},{"code_sha256_prefix":"95ea327200d6a5f0","entry":"extract_json_data","repo":"declare-lab/della","repo_kind":"official","path":"gpt4_as_judge_gsm8k.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/gpt4_as_judge_gsm8k.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"95ea327200d6a5f0"}},{"code_sha256_prefix":"56f282445e93d633","entry":"extract_json_labels","repo":"declare-lab/della","repo_kind":"official","path":"gpt4_as_judge_gsm8k.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/gpt4_as_judge_gsm8k.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56f282445e93d633"}},{"code_sha256_prefix":"97f0bf7971c7c44c","entry":"generate_code_task_prompt","repo":"declare-lab/della","repo_kind":"official","path":"utils/evaluate_llms_utils.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/utils/evaluate_llms_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"97f0bf7971c7c44c"}},{"code_sha256_prefix":"d18e070ddca442ea","entry":"generate_instruction_following_task_prompt","repo":"declare-lab/della","repo_kind":"official","path":"utils/evaluate_llms_utils.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/utils/evaluate_llms_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d18e070ddca442ea"}},{"code_sha256_prefix":"69bebec3d08ead38","entry":"get_mask","repo":"declare-lab/della","repo_kind":"official","path":"mergekit/mergekit/merge_methods/generalized_task_arithmetic.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/mergekit/mergekit/merge_methods/generalized_task_arithmetic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69bebec3d08ead38"}},{"code_sha256_prefix":"bf298728b02d6a49","entry":"method_md","repo":"declare-lab/della","repo_kind":"official","path":"mergekit/mergekit/card.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/mergekit/mergekit/card.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf298728b02d6a49"}},{"code_sha256_prefix":"9236f9e6f951cf70","entry":"parse_kmb","repo":"declare-lab/della","repo_kind":"official","path":"mergekit/mergekit/common.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/mergekit/mergekit/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9236f9e6f951cf70"}},{"code_sha256_prefix":"bf391789c976e56c","entry":"take_common_submatrix","repo":"declare-lab/della","repo_kind":"official","path":"mergekit/mergekit/common.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/mergekit/mergekit/common.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf391789c976e56c"}},{"code_sha256_prefix":"d49412b3a690206e","entry":"batch_data","repo":"declare-lab/della","repo_kind":"official","path":"utils/evaluate_llms_utils.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/utils/evaluate_llms_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d49412b3a690206e"}},{"code_sha256_prefix":"6f7a509088909eaa","entry":"batch_response_gen","repo":"declare-lab/della","repo_kind":"official","path":"generate.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f7a509088909eaa"}},{"code_sha256_prefix":"e3ab744bf540c3b8","entry":"chat_completion","repo":"declare-lab/della","repo_kind":"official","path":"gpt4_as_judge_gsm8k.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/gpt4_as_judge_gsm8k.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3ab744bf540c3b8"}},{"code_sha256_prefix":"8d286435d41a4165","entry":"is_hf","repo":"declare-lab/della","repo_kind":"official","path":"mergekit/mergekit/card.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/mergekit/mergekit/card.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d286435d41a4165"}},{"code_sha256_prefix":"8f325d88a9965387","entry":"maybe_link_hf","repo":"declare-lab/della","repo_kind":"official","path":"mergekit/mergekit/card.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/mergekit/mergekit/card.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f325d88a9965387"}},{"code_sha256_prefix":"5c23dd917eee4f4d","entry":"response_gen","repo":"declare-lab/della","repo_kind":"official","path":"generate.py","file_url":"https://github.com/declare-lab/della/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c23dd917eee4f4d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}