{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sleb-streamlining-llms-through-redundancy","title":"SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks","arxiv_id":"2402.09025","date":"2024-02-14","proceeding":null,"authors":["Jiwon Song","Kyungseok Oh","Taesu Kim","HyungJun Kim","Yulhwa Kim","Jae-Joon Kim"],"abstract":"Large language models (LLMs) have proven to be highly effective across various natural language processing tasks. However, their large number of parameters poses significant challenges for practical deployment. Pruning, a technique aimed at reducing the size and complexity of LLMs, offers a potential solution by removing redundant components from the network. Despite the promise of pruning, existing methods often struggle to achieve substantial end-to-end LLM inference speedup. In this paper, we introduce SLEB, a novel approach designed to streamline LLMs by eliminating redundant transformer blocks. We choose the transformer block as the fundamental unit for pruning, because LLMs exhibit block-level redundancy with high similarity between the outputs of neighboring blocks. This choice allows us to effectively enhance the processing speed of LLMs. Our experimental results demonstrate that SLEB outperforms previous LLM pruning methods in accelerating LLM inference while also maintaining superior perplexity and accuracy, making SLEB as a promising technique for enhancing the efficiency of LLMs. The code is available at: https://github.com/jiwonsong-dev/SLEB.","url_abs":"https://arxiv.org/abs/2402.09025v6","url_pdf":"https://arxiv.org/pdf/2402.09025v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sleb-streamlining-llms-through-redundancy","repo_url":"https://github.com/jiwonsong-dev/sleb","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.09025","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.09025"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiwonsong-dev/sleb","reach":null}],"summary":{"ran_draft_wrong":3,"ran_honours":3,"unverified":6},"by_repo_kind":{"official":{"samples":12,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"97b79932a67ad62d","entry":"block_remove","repo":"jiwonsong-dev/sleb","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/sleb/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97b79932a67ad62d"}},{"code_sha256_prefix":"7a2e338ff7fb21b0","entry":"block_remove_llama","repo":"jiwonsong-dev/sleb","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/sleb/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a2e338ff7fb21b0"}},{"code_sha256_prefix":"62cb0034e375daea","entry":"block_remove_opt","repo":"jiwonsong-dev/sleb","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/sleb/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62cb0034e375daea"}},{"code_sha256_prefix":"a7e0158215732c17","entry":"eval_ppl","repo":"jiwonsong-dev/sleb","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/sleb/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7e0158215732c17"}},{"code_sha256_prefix":"084002d7f8a76578","entry":"get_loss","repo":"jiwonsong-dev/SLEB","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/SLEB/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"084002d7f8a76578"}},{"code_sha256_prefix":"ece3a1bcea1d0f62","entry":"load_and_eval_ppl","repo":"jiwonsong-dev/sleb","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/sleb/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ece3a1bcea1d0f62"}},{"code_sha256_prefix":"490bf0726aca1461","entry":"block_replace","repo":"jiwonsong-dev/sleb","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/sleb/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"490bf0726aca1461"}},{"code_sha256_prefix":"01995feb8be8923d","entry":"eval_zero_shot","repo":"jiwonsong-dev/sleb","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/sleb/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01995feb8be8923d"}},{"code_sha256_prefix":"3d35b1f3a44e3c8d","entry":"get_llm","repo":"jiwonsong-dev/sleb","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/sleb/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d35b1f3a44e3c8d"}},{"code_sha256_prefix":"acd45bf0eb9cb9f4","entry":"sleb","repo":"jiwonsong-dev/sleb","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/sleb/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"acd45bf0eb9cb9f4"}},{"code_sha256_prefix":"cb61b072985a3343","entry":"turn_off","repo":"jiwonsong-dev/sleb","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/sleb/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb61b072985a3343"}},{"code_sha256_prefix":"19d5226fd25d03fc","entry":"turn_on","repo":"jiwonsong-dev/sleb","repo_kind":"official","path":"sleb.py","file_url":"https://github.com/jiwonsong-dev/sleb/blob/HEAD/sleb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19d5226fd25d03fc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}