{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kangaroo-lossless-self-speculative-decoding","title":"Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting","arxiv_id":"2404.18911","date":"2024-04-29","proceeding":null,"authors":["Fangcheng Liu","Yehui Tang","Zhenhua Liu","Yunsheng Ni","Kai Han","Yunhe Wang"],"abstract":"Speculative decoding has demonstrated its effectiveness in accelerating the inference of large language models while maintaining a consistent sampling distribution. However, the conventional approach of training a separate draft model to achieve a satisfactory token acceptance rate can be costly. Drawing inspiration from early exiting, we propose a novel self-speculative decoding framework \\emph{Kangaroo}, which uses a fixed shallow sub-network as a self-draft model, with the remaining layers serving as the larger target model. We train a lightweight and efficient adapter module on top of the sub-network to bridge the gap between the sub-network and the full model's representation ability. It is noteworthy that the inference latency of the self-draft model may no longer be negligible compared to the large model, necessitating strategies to increase the token acceptance rate while minimizing the drafting steps of the small model. To address this challenge, we introduce an additional early exiting mechanism for generating draft tokens. Specifically, we halt the small model's subsequent prediction during the drafting phase once the confidence level for the current token falls below a certain threshold. Extensive experiments on the Spec-Bench demonstrate the effectiveness of Kangaroo. Under single-sequence verification, Kangaroo achieves speedups up to $1.68\\times$ on Spec-Bench, outperforming Medusa-1 with 88.7\\% fewer additional parameters (67M compared to 591M). The code for Kangaroo is available at https://github.com/Equationliu/Kangaroo.","url_abs":"https://arxiv.org/abs/2404.18911v1","url_pdf":"https://arxiv.org/pdf/2404.18911v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kangaroo-lossless-self-speculative-decoding","repo_url":"https://github.com/Equationliu/Kangaroo","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"early-exiting","method_name":"Early exiting"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.18911","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.18911"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Equationliu/Kangaroo","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"ran":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"Equationliu/Kangaroo","repo_kind":"official","path":"kangaroo/adapter.py","file_url":"https://github.com/Equationliu/Kangaroo/blob/HEAD/kangaroo/adapter.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"Equationliu/Kangaroo","repo_kind":"official","path":"kangaroo/adapter.py","file_url":"https://github.com/Equationliu/Kangaroo/blob/HEAD/kangaroo/adapter.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"df4cc783cc8f5272","entry":"baseline_forward","repo":"Equationliu/Kangaroo","repo_kind":"official","path":"evaluation/inference_baseline.py","file_url":"https://github.com/Equationliu/Kangaroo/blob/HEAD/evaluation/inference_baseline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df4cc783cc8f5272"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"Equationliu/Kangaroo","repo_kind":"official","path":"kangaroo/adapter.py","file_url":"https://github.com/Equationliu/Kangaroo/blob/HEAD/kangaroo/adapter.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"59f5346ce1d6bf08","entry":"kangaroo_forward","repo":"Equationliu/Kangaroo","repo_kind":"official","path":"evaluation/inference_kangaroo.py","file_url":"https://github.com/Equationliu/Kangaroo/blob/HEAD/evaluation/inference_kangaroo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"59f5346ce1d6bf08"}},{"code_sha256_prefix":"aac7165992097239","entry":"speed","repo":"Equationliu/Kangaroo","repo_kind":"official","path":"evaluation/speed.py","file_url":"https://github.com/Equationliu/Kangaroo/blob/HEAD/evaluation/speed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aac7165992097239"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}