{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dare-the-extreme-revisiting-delta-parameter","title":"DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models","arxiv_id":"2410.09344","date":"2024-10-12","proceeding":null,"authors":["Wenlong Deng","Yize Zhao","Vala Vakilian","Minghui Chen","Xiaoxiao Li","Christos Thrampoulidis"],"abstract":"Storing open-source fine-tuned models separately introduces redundancy and increases response times in applications utilizing multiple models. Delta-parameter pruning (DPP), particularly the random drop and rescale (DARE) method proposed by Yu et al., addresses this by pruning the majority of delta parameters--the differences between fine-tuned and pre-trained model weights--while typically maintaining minimal performance loss. However, DARE fails when either the pruning rate or the magnitude of the delta parameters is large. We highlight two key reasons for this failure: (1) an excessively large rescaling factor as pruning rates increase, and (2) high mean and variance in the delta parameters. To push DARE's limits, we introduce DAREx (DARE the eXtreme), which features two algorithmic improvements: (1) DAREx-q, a rescaling factor modification that significantly boosts performance at high pruning rates (e.g., >30 % on COLA and SST2 for encoder models, with even greater gains in decoder models), and (2) DAREx-L2, which combines DARE with AdamR, an in-training method that applies appropriate delta regularization before DPP. We also demonstrate that DAREx-q can be seamlessly combined with vanilla parameter-efficient fine-tuning techniques like LoRA and can facilitate structural DPP. Additionally, we revisit the application of importance-based pruning techniques within DPP, demonstrating that they outperform random-based methods when delta parameters are large. Through this comprehensive study, we develop a pipeline for selecting the most appropriate DPP method under various practical scenarios.","url_abs":"https://arxiv.org/abs/2410.09344v2","url_pdf":"https://arxiv.org/pdf/2410.09344v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dare-the-extreme-revisiting-delta-parameter","repo_url":"https://github.com/vengdeng/darex","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"CoLA"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"cola","method_name":"COLA"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.09344","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.09344"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vengdeng/darex","reach":{"status":"ok"}}],"summary":{"ran_violates":1,"ran":10,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":17,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"0fe9cf6c00ef56ea","entry":"is_number","repo":"vengdeng/darex","repo_kind":"official","path":"utils/evaluate_llms_utils.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/evaluate_llms_utils.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fe9cf6c00ef56ea"}},{"code_sha256_prefix":"f610ce38c215e0b3","entry":"add_back_decoder","repo":"vengdeng/darex","repo_kind":"official","path":"utils/deltaprune.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/deltaprune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f610ce38c215e0b3"}},{"code_sha256_prefix":"d49a90d96daf66f7","entry":"batch_data","repo":"vengdeng/darex","repo_kind":"official","path":"find_q_decoder.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/find_q_decoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d49a90d96daf66f7"}},{"code_sha256_prefix":"e3f5f65ef9c76959","entry":"cal_delta_param_decoder","repo":"vengdeng/darex","repo_kind":"official","path":"utils/deltaprune.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/deltaprune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3f5f65ef9c76959"}},{"code_sha256_prefix":"d877155389b89d32","entry":"drop_decoder","repo":"vengdeng/darex","repo_kind":"official","path":"utils/deltaprune.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/deltaprune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d877155389b89d32"}},{"code_sha256_prefix":"95693ad4a192c071","entry":"extract_res","repo":"vengdeng/darex","repo_kind":"official","path":"utils/metrics.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"95693ad4a192c071"}},{"code_sha256_prefix":"2ed16fcdc14ad951","entry":"find_layers","repo":"vengdeng/darex","repo_kind":"official","path":"utils/getwx.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/getwx.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ed16fcdc14ad951"}},{"code_sha256_prefix":"aa111ae3259d74f6","entry":"get_outpuchage","repo":"vengdeng/darex","repo_kind":"official","path":"utils/outputchange.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/outputchange.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa111ae3259d74f6"}},{"code_sha256_prefix":"23f6861e67c10b94","entry":"optimal_q_calculation","repo":"vengdeng/darex","repo_kind":"official","path":"utils/analyticalq.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/analyticalq.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"23f6861e67c10b94"}},{"code_sha256_prefix":"6ea7104a43c76889","entry":"output_loader","repo":"vengdeng/darex","repo_kind":"official","path":"utils/outputchange.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/outputchange.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ea7104a43c76889"}},{"code_sha256_prefix":"87cb5010447732ab","entry":"phi_func","repo":"vengdeng/darex","repo_kind":"official","path":"utils/analyticalq.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/analyticalq.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87cb5010447732ab"}},{"code_sha256_prefix":"db6d55cd1fb1ce20","entry":"remove_one_outlier","repo":"vengdeng/darex","repo_kind":"official","path":"find_q_encoder.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/find_q_encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db6d55cd1fb1ce20"}},{"code_sha256_prefix":"080b98d73a404601","entry":"analytical_resolve_new","repo":"vengdeng/darex","repo_kind":"official","path":"utils/analyticalq.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/analyticalq.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"080b98d73a404601"}},{"code_sha256_prefix":"d49412b3a690206e","entry":"batch_data","repo":"vengdeng/darex","repo_kind":"official","path":"utils/evaluate_llms_utils.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/evaluate_llms_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d49412b3a690206e"}},{"code_sha256_prefix":"a54651cc8454cb77","entry":"get_loaders","repo":"vengdeng/darex","repo_kind":"official","path":"utils/getwx.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/getwx.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a54651cc8454cb77"}},{"code_sha256_prefix":"6839d02854b154b2","entry":"get_outpuchage_decoder","repo":"vengdeng/darex","repo_kind":"official","path":"utils/outputchange.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/outputchange.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6839d02854b154b2"}},{"code_sha256_prefix":"ecc04d6edadf3e8b","entry":"get_wikitext2","repo":"vengdeng/darex","repo_kind":"official","path":"utils/getwx.py","file_url":"https://github.com/vengdeng/darex/blob/HEAD/utils/getwx.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ecc04d6edadf3e8b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}