{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/shortened-llama-a-simple-depth-pruning-for","title":"Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods","arxiv_id":"2402.02834","date":"2024-02-05","proceeding":null,"authors":["Bo-Kyeong Kim","Geonmin Kim","Tae-Ho Kim","Thibault Castells","Shinkook Choi","Junho Shin","Hyoung-Kyu Song"],"abstract":"Structured pruning of modern large language models (LLMs) has emerged as a way of decreasing their high computational needs. Width pruning reduces the size of projection weight matrices (e.g., by removing attention heads) while maintaining the number of layers. Depth pruning, in contrast, removes entire layers or blocks, while keeping the size of the remaining weights unchanged. Most current research focuses on either width-only or a blend of width and depth pruning, with little comparative analysis between the two units (width vs. depth) concerning their impact on LLM inference efficiency. In this work, we show that simple depth pruning can effectively compress LLMs while achieving comparable or superior performance to recent width pruning studies. Our pruning method boosts inference speeds, especially under memory-constrained conditions that require limited batch sizes for running LLMs, where width pruning is ineffective. In retraining pruned models for quality recovery, continued pretraining on a large corpus markedly outperforms LoRA-based tuning, particularly at severe pruning ratios. We hope this work can help build compact yet capable LLMs. Code and models can be found at: https://github.com/Nota-NetsPresso/shortened-llm","url_abs":"https://arxiv.org/abs/2402.02834v2","url_pdf":"https://arxiv.org/pdf/2402.02834v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"shortened-llama-a-simple-depth-pruning-for","repo_url":"https://github.com/nota-netspresso/shortened-llm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.02834","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.02834"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nota-netspresso/shortened-llm","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"32380a050ca033d4","entry":"b2mb","repo":"nota-netspresso/shortened-llm","repo_kind":"official","path":"src/gen_batch_eval_vram.py","file_url":"https://github.com/nota-netspresso/shortened-llm/blob/HEAD/src/gen_batch_eval_vram.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32380a050ca033d4"}},{"code_sha256_prefix":"9012ddbbaf4ee64f","entry":"llama_eval","repo":"nota-netspresso/shortened-llm","repo_kind":"official","path":"src/eval_ppl.py","file_url":"https://github.com/nota-netspresso/shortened-llm/blob/HEAD/src/eval_ppl.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9012ddbbaf4ee64f"}},{"code_sha256_prefix":"8a69aa6e08dc65a9","entry":"process_data","repo":"nota-netspresso/shortened-llm","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/nota-netspresso/shortened-llm/blob/HEAD/src/dataset.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a69aa6e08dc65a9"}},{"code_sha256_prefix":"dda7f6122e4d1904","entry":"get_examples","repo":"nota-netspresso/shortened-llm","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/nota-netspresso/shortened-llm/blob/HEAD/src/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dda7f6122e4d1904"}},{"code_sha256_prefix":"f8178f1b0c33cf48","entry":"get_gpu_metrics","repo":"nota-netspresso/shortened-llm","repo_kind":"official","path":"src/gen_batch_eval_time_gpuutil.py","file_url":"https://github.com/nota-netspresso/shortened-llm/blob/HEAD/src/gen_batch_eval_time_gpuutil.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8178f1b0c33cf48"}},{"code_sha256_prefix":"4ef386ce30344069","entry":"get_loaders","repo":"nota-netspresso/shortened-llm","repo_kind":"official","path":"src/dataset.py","file_url":"https://github.com/nota-netspresso/shortened-llm/blob/HEAD/src/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ef386ce30344069"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}