{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dynamic-sparse-no-training-training-free-fine","title":"Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs","arxiv_id":"2310.08915","date":"2023-10-13","proceeding":null,"authors":["Yuxin Zhang","Lirui Zhao","Mingbao Lin","Yunyun Sun","Yiwu Yao","Xingjia Han","Jared Tanner","Shiwei Liu","Rongrong Ji"],"abstract":"The ever-increasing large language models (LLMs), though opening a potential path for the upcoming artificial general intelligence, sadly drops a daunting obstacle on the way towards their on-device deployment. As one of the most well-established pre-LLMs approaches in reducing model complexity, network pruning appears to lag behind in the era of LLMs, due mostly to its costly fine-tuning (or re-training) necessity under the massive volumes of model parameter and training data. To close this industry-academia gap, we introduce Dynamic Sparse No Training (DSnoT), a training-free fine-tuning approach that slightly updates sparse LLMs without the expensive backpropagation and any weight updates. Inspired by the Dynamic Sparse Training, DSnoT minimizes the reconstruction error between the dense and sparse LLMs, in the fashion of performing iterative weight pruning-and-growing on top of sparse LLMs. To accomplish this purpose, DSnoT particularly takes into account the anticipated reduction in reconstruction error for pruning and growing, as well as the variance w.r.t. different input data for growing each weight. This practice can be executed efficiently in linear time since its obviates the need of backpropagation for fine-tuning LLMs. Extensive experiments on LLaMA-V1/V2, Vicuna, and OPT across various benchmarks demonstrate the effectiveness of DSnoT in enhancing the performance of sparse LLMs, especially at high sparsity levels. For instance, DSnoT is able to outperform the state-of-the-art Wanda by 26.79 perplexity at 70% sparsity with LLaMA-7B. Our paper offers fresh insights into how to fine-tune sparse LLMs in an efficient training-free manner and open new venues to scale the great potential of sparsity to LLMs. Codes are available at https://github.com/zyxxmu/DSnoT.","url_abs":"https://arxiv.org/abs/2310.08915v3","url_pdf":"https://arxiv.org/pdf/2310.08915v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dynamic-sparse-no-training-training-free-fine","repo_url":"https://github.com/zyxxmu/dsnot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"network-pruning","task_name":"Network Pruning"}],"methods":[{"method_slug":"opt","method_name":"OPT"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2310.08915","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.08915"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zyxxmu/DSnoT","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zyxxmu/dsnot","reach":{"status":"ok"}}],"summary":{"ran":4,"ran_honours":2,"ran_draft_wrong":2,"unverified":7},"by_repo_kind":{"official":{"samples":15,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"37d49ff5ade0a79c","entry":"TokenizerWrapper","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"37d49ff5ade0a79c"}},{"code_sha256_prefix":"ba57808cc18062f0","entry":"WrappedGPT","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba57808cc18062f0"}},{"code_sha256_prefix":"75c1b08e37a25564","entry":"check_sparsity","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"75c1b08e37a25564"}},{"code_sha256_prefix":"2ed16fcdc14ad951","entry":"find_layers","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ed16fcdc14ad951"}},{"code_sha256_prefix":"dee8be97b2cf4692","entry":"get_wikitext2","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dee8be97b2cf4692"}},{"code_sha256_prefix":"0d1f4e924ac02ca3","entry":"prepare_calibration_input","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d1f4e924ac02ca3"}},{"code_sha256_prefix":"b12425287b2eaa77","entry":"prepare_calibration_input_opt","repo":"zyxxmu/DSnoT","repo_kind":"official","path":"lib/prune_opt.py","file_url":"https://github.com/zyxxmu/DSnoT/blob/HEAD/lib/prune_opt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b12425287b2eaa77"}},{"code_sha256_prefix":"e9701291631dbba0","entry":"return_reorder_indice","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9701291631dbba0"}},{"code_sha256_prefix":"939b3d0967327424","entry":"check_sparsity_opt","repo":"zyxxmu/DSnoT","repo_kind":"official","path":"lib/prune_opt.py","file_url":"https://github.com/zyxxmu/DSnoT/blob/HEAD/lib/prune_opt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"939b3d0967327424"}},{"code_sha256_prefix":"7630a157a3b32ab9","entry":"get_c4","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7630a157a3b32ab9"}},{"code_sha256_prefix":"2551834c320a844b","entry":"get_loaders","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2551834c320a844b"}},{"code_sha256_prefix":"3b99c66b488c6dc0","entry":"get_ptb","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b99c66b488c6dc0"}},{"code_sha256_prefix":"65c1747a45c2f32d","entry":"prune_DSnoT","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65c1747a45c2f32d"}},{"code_sha256_prefix":"b4d0a32a2f3c05ee","entry":"prune_DSnoT_opt","repo":"zyxxmu/DSnoT","repo_kind":"official","path":"lib/prune_opt.py","file_url":"https://github.com/zyxxmu/DSnoT/blob/HEAD/lib/prune_opt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4d0a32a2f3c05ee"}},{"code_sha256_prefix":"dd62cd1d05f3dff8","entry":"save_time_result","repo":"zyxxmu/dsnot","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/zyxxmu/dsnot/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dd62cd1d05f3dff8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}