{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-pruning-large-language-models","title":"Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization","arxiv_id":"2406.15524","date":"2024-06-21","proceeding":null,"authors":["Sungbin Shin","Wonpyo Park","Jaeho Lee","Namhoon Lee"],"abstract":"This work suggests fundamentally rethinking the current practice of pruning large language models (LLMs). The way it is done is by divide and conquer: split the model into submodels, sequentially prune them, and reconstruct predictions of the dense counterparts on small calibration data one at a time; the final model is obtained simply by putting the resulting sparse submodels together. While this approach enables pruning under memory constraints, it generates high reconstruction errors. In this work, we first present an array of reconstruction techniques that can significantly reduce this error by more than $90\\%$. Unwittingly, however, we discover that minimizing reconstruction error is not always ideal and can overfit the given calibration data, resulting in rather increased language perplexity and poor performance at downstream tasks. We find out that a strategy of self-generating calibration data can mitigate this trade-off between reconstruction and generalization, suggesting new directions in the presence of both benefits and pitfalls of reconstruction for pruning LLMs.","url_abs":"https://arxiv.org/abs/2406.15524v2","url_pdf":"https://arxiv.org/pdf/2406.15524v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-pruning-large-language-models","repo_url":"https://github.com/log-postech/rethinking-llm-pruning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.15524","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.15524"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/log-postech/rethinking-llm-pruning","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":5},"by_repo_kind":{"official":{"samples":8,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"dee8be97b2cf4692","entry":"get_wikitext2","repo":"log-postech/rethinking-llm-pruning","repo_kind":"official","path":"lib/data.py","file_url":"https://github.com/log-postech/rethinking-llm-pruning/blob/HEAD/lib/data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dee8be97b2cf4692"}},{"code_sha256_prefix":"4daf7035d84ce598","entry":"loss_fn","repo":"log-postech/rethinking-llm-pruning","repo_kind":"official","path":"lib/llama/finetune.py","file_url":"https://github.com/log-postech/rethinking-llm-pruning/blob/HEAD/lib/llama/finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4daf7035d84ce598"}},{"code_sha256_prefix":"9f9732515acc7a53","entry":"obtain_output","repo":"log-postech/rethinking-llm-pruning","repo_kind":"official","path":"lib/llama/finetune.py","file_url":"https://github.com/log-postech/rethinking-llm-pruning/blob/HEAD/lib/llama/finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9f9732515acc7a53"}},{"code_sha256_prefix":"112fa6e5a362cc14","entry":"get_c4","repo":"log-postech/rethinking-llm-pruning","repo_kind":"official","path":"lib/data.py","file_url":"https://github.com/log-postech/rethinking-llm-pruning/blob/HEAD/lib/data.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"112fa6e5a362cc14"}},{"code_sha256_prefix":"bea08127e3c1ac0f","entry":"get_llm","repo":"log-postech/rethinking-llm-pruning","repo_kind":"official","path":"opt.py","file_url":"https://github.com/log-postech/rethinking-llm-pruning/blob/HEAD/opt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bea08127e3c1ac0f"}},{"code_sha256_prefix":"74f756783cc6ae2c","entry":"get_llm_fp16","repo":"log-postech/rethinking-llm-pruning","repo_kind":"official","path":"llama.py","file_url":"https://github.com/log-postech/rethinking-llm-pruning/blob/HEAD/llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74f756783cc6ae2c"}},{"code_sha256_prefix":"3b99c66b488c6dc0","entry":"get_ptb","repo":"log-postech/rethinking-llm-pruning","repo_kind":"official","path":"lib/data.py","file_url":"https://github.com/log-postech/rethinking-llm-pruning/blob/HEAD/lib/data.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b99c66b488c6dc0"}},{"code_sha256_prefix":"c8b35875f810484d","entry":"val","repo":"log-postech/rethinking-llm-pruning","repo_kind":"official","path":"lib/llama/finetune.py","file_url":"https://github.com/log-postech/rethinking-llm-pruning/blob/HEAD/lib/llama/finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8b35875f810484d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}