{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/besa-pruning-large-language-models-with","title":"BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation","arxiv_id":"2402.16880","date":"2024-02-18","proceeding":null,"authors":["Peng Xu","Wenqi Shao","Mengzhao Chen","Shitao Tang","Kaipeng Zhang","Peng Gao","Fengwei An","Yu Qiao","Ping Luo"],"abstract":"Large language models (LLMs) have demonstrated outstanding performance in various tasks, such as text summarization, text question-answering, and etc. While their performance is impressive, the computational footprint due to their vast number of parameters can be prohibitive. Existing solutions such as SparseGPT and Wanda attempt to alleviate this issue through weight pruning. However, their layer-wise approach results in significant perturbation to the model's output and requires meticulous hyperparameter tuning, such as the pruning rate, which can adversely affect overall model performance. To address this, this paper introduces a novel LLM pruning technique dubbed blockwise parameter-efficient sparsity allocation (BESA) by applying a blockwise reconstruction loss. In contrast to the typical layer-wise pruning techniques, BESA is characterized by two distinctive attributes: i) it targets the overall pruning error with respect to individual transformer blocks, and ii) it allocates layer-specific sparsity in a differentiable manner, both of which ensure reduced performance degradation after pruning. Our experiments show that BESA achieves state-of-the-art performance, efficiently pruning LLMs like LLaMA1, and LLaMA2 with 7B to 70B parameters on a single A100 GPU in just five hours. Code is available at https://github.com/OpenGVLab/LLMPrune-BESA.","url_abs":"https://arxiv.org/abs/2402.16880v2","url_pdf":"https://arxiv.org/pdf/2402.16880v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"besa-pruning-large-language-models-with","repo_url":"https://github.com/linkanonymous/besa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"besa-pruning-large-language-models-with","repo_url":"https://github.com/opengvlab/llmprune-besa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"text-summarization","task_name":"Text Summarization"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.16880","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.16880"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/linkanonymous/besa","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/llmprune-besa","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_draft_wrong":2,"unverified":6},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"65c00a6635e49555","entry":"calc_sparsity","repo":"opengvlab/llmprune-besa","repo_kind":"official","path":"hw_sim/utils.py","file_url":"https://github.com/opengvlab/llmprune-besa/blob/HEAD/hw_sim/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65c00a6635e49555"}},{"code_sha256_prefix":"14fab8e91e876c60","entry":"find_layers","repo":"linkanonymous/besa","repo_kind":"official","path":"utils/tools.py","file_url":"https://github.com/linkanonymous/besa/blob/HEAD/utils/tools.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14fab8e91e876c60"}},{"code_sha256_prefix":"c12b55ef635e9947","entry":"pattern_match","repo":"linkanonymous/besa","repo_kind":"official","path":"utils/tools.py","file_url":"https://github.com/linkanonymous/besa/blob/HEAD/utils/tools.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c12b55ef635e9947"}},{"code_sha256_prefix":"d5bc1562a14aa7f1","entry":"sparse_linear_flops","repo":"opengvlab/llmprune-besa","repo_kind":"official","path":"hw_sim/ViTCoD_comp.py","file_url":"https://github.com/opengvlab/llmprune-besa/blob/HEAD/hw_sim/ViTCoD_comp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d5bc1562a14aa7f1"}},{"code_sha256_prefix":"51a53fa33d2d9a6f","entry":"get_c4","repo":"linkanonymous/besa","repo_kind":"official","path":"utils/data.py","file_url":"https://github.com/linkanonymous/besa/blob/HEAD/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"51a53fa33d2d9a6f"}},{"code_sha256_prefix":"e19c4cba75bdc00a","entry":"get_model","repo":"opengvlab/llmprune-besa","repo_kind":"official","path":"hw_sim/utils.py","file_url":"https://github.com/opengvlab/llmprune-besa/blob/HEAD/hw_sim/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e19c4cba75bdc00a"}},{"code_sha256_prefix":"3c22f52a4d22d5d2","entry":"get_ptb","repo":"linkanonymous/besa","repo_kind":"official","path":"utils/data.py","file_url":"https://github.com/linkanonymous/besa/blob/HEAD/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c22f52a4d22d5d2"}},{"code_sha256_prefix":"a8a15ad5400c13dc","entry":"get_wikitext2","repo":"linkanonymous/besa","repo_kind":"official","path":"utils/data.py","file_url":"https://github.com/linkanonymous/besa/blob/HEAD/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8a15ad5400c13dc"}},{"code_sha256_prefix":"eb6549618213eabe","entry":"process_sparse_layer","repo":"opengvlab/llmprune-besa","repo_kind":"official","path":"hw_sim/utils.py","file_url":"https://github.com/opengvlab/llmprune-besa/blob/HEAD/hw_sim/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eb6549618213eabe"}},{"code_sha256_prefix":"582d1cd9c741708f","entry":"sparse_linear_simulate","repo":"opengvlab/llmprune-besa","repo_kind":"official","path":"hw_sim/ViTCoD_comp.py","file_url":"https://github.com/opengvlab/llmprune-besa/blob/HEAD/hw_sim/ViTCoD_comp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"582d1cd9c741708f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}