{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/layer-skip-enabling-early-exit-inference-and","title":"LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding","arxiv_id":"2404.16710","date":"2024-04-25","proceeding":null,"authors":["Mostafa Elhoushi","Akshat Shrivastava","Diana Liskovich","Basil Hosmer","Bram Wasti","Liangzhen Lai","Anas Mahmoud","Bilge Acun","Saurabh Agarwal","Ahmed Roman","Ahmed A Aly","Beidi Chen","Carole-Jean Wu"],"abstract":"We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and checkpoints at https://github.com/facebookresearch/LayerSkip.","url_abs":"https://arxiv.org/abs/2404.16710v4","url_pdf":"https://arxiv.org/pdf/2404.16710v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"layer-skip-enabling-early-exit-inference-and","repo_url":"https://github.com/facebookresearch/layerskip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"hellaswag","task_name":"HellaSwag"},{"task_slug":"humaneval","task_name":"HumanEval"},{"task_slug":"mmlu","task_name":"MMLU"},{"task_slug":"math","task_name":"Math"},{"task_slug":"natural-questions","task_name":"Natural Questions"},{"task_slug":"race-h","task_name":"RACE-h"},{"task_slug":"race-m","task_name":"RACE-m"},{"task_slug":"triviaqa","task_name":"TriviaQA"},{"task_slug":"winogrande","task_name":"Winogrande"}],"methods":[{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"early-exiting","method_name":"Early exiting"},{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"layerdrop","method_name":"LayerDrop"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.16710","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.16710"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/layerskip","reach":null}],"summary":{"ran":1,"ran_draft_wrong":1,"ran_fixture":2,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"523cc4a218246a99","entry":"ForwardResult","repo":"facebookresearch/layerskip","repo_kind":"official","path":"self_speculation/llama_model_utils.py","file_url":"https://github.com/facebookresearch/layerskip/blob/HEAD/self_speculation/llama_model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"523cc4a218246a99"}},{"code_sha256_prefix":"fc6b3d6f55e1835e","entry":"_expand_mask","repo":"facebookresearch/layerskip","repo_kind":"official","path":"self_speculation/llama_model_utils.py","file_url":"https://github.com/facebookresearch/layerskip/blob/HEAD/self_speculation/llama_model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fc6b3d6f55e1835e"}},{"code_sha256_prefix":"72d28356d328b1e1","entry":"_make_causal_mask","repo":"facebookresearch/layerskip","repo_kind":"official","path":"self_speculation/llama_model_utils.py","file_url":"https://github.com/facebookresearch/layerskip/blob/HEAD/self_speculation/llama_model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"72d28356d328b1e1"}},{"code_sha256_prefix":"bdb174316c821be9","entry":"_prepare_decoder_attention_mask","repo":"facebookresearch/layerskip","repo_kind":"official","path":"self_speculation/llama_model_utils.py","file_url":"https://github.com/facebookresearch/layerskip/blob/HEAD/self_speculation/llama_model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bdb174316c821be9"}},{"code_sha256_prefix":"0b97715e46a835ff","entry":"forward_early","repo":"facebookresearch/layerskip","repo_kind":"official","path":"self_speculation/llama_model_utils.py","file_url":"https://github.com/facebookresearch/layerskip/blob/HEAD/self_speculation/llama_model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0b97715e46a835ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}