{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ouroboros-speculative-decoding-with-large","title":"Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding","arxiv_id":"2402.13720","date":"2024-02-21","proceeding":null,"authors":["Weilin Zhao","Yuxiang Huang","Xu Han","Wang Xu","Chaojun Xiao","Xinrong Zhang","Yewei Fang","Kaihuo Zhang","Zhiyuan Liu","Maosong Sun"],"abstract":"Speculative decoding is a widely used method that accelerates the generation process of large language models (LLMs) with no compromise in model performance. It achieves this goal by using an existing smaller model for drafting and then employing the target LLM to verify the draft in a low-cost parallel manner. Under such a drafting-verification framework, drafting efficiency has become a bottleneck in the final speedup of speculative decoding. Therefore, generating longer drafts at less cost can lead to better decoding speedup. To achieve this, we introduce Ouroboros, which can generate draft phrases to parallelize the drafting process and meanwhile lengthen drafts in a training-free manner. The experimental results on various typical text generation tasks show that Ouroboros can achieve speedups of up to $2.8\\times$ over speculative decoding and $3.9\\times$ over vanilla decoding, without fine-tuning draft and target models. The source code of Ouroboros is available at https://github.com/thunlp/Ouroboros.","url_abs":"https://arxiv.org/abs/2402.13720v3","url_pdf":"https://arxiv.org/pdf/2402.13720v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ouroboros-speculative-decoding-with-large","repo_url":"https://github.com/thunlp/ouroboros","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.13720","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.13720"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp/ouroboros","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/thunlp/Ouroboros","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran":3,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":11,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"thunlp/ouroboros","repo_kind":"official","path":"ouroboros/models/modeling_llama.py","file_url":"https://github.com/thunlp/ouroboros/blob/HEAD/ouroboros/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"6a1b1dc5e77396d7","entry":"CacheEngine","repo":"thunlp/Ouroboros","repo_kind":"official","path":"ouroboros/ouroboros.py","file_url":"https://github.com/thunlp/Ouroboros/blob/HEAD/ouroboros/ouroboros.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6a1b1dc5e77396d7"}},{"code_sha256_prefix":"20e375bb284e9206","entry":"KVCacheModelLade","repo":"thunlp/Ouroboros","repo_kind":"official","path":"ouroboros/ouroboros.py","file_url":"https://github.com/thunlp/Ouroboros/blob/HEAD/ouroboros/ouroboros.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20e375bb284e9206"}},{"code_sha256_prefix":"d72d77c358c2f8ad","entry":"KVCacheModelSimpleWithGuess","repo":"thunlp/Ouroboros","repo_kind":"official","path":"ouroboros/ouroboros.py","file_url":"https://github.com/thunlp/Ouroboros/blob/HEAD/ouroboros/ouroboros.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d72d77c358c2f8ad"}},{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"thunlp/ouroboros","repo_kind":"official","path":"ouroboros/models/modeling_llama.py","file_url":"https://github.com/thunlp/ouroboros/blob/HEAD/ouroboros/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"thunlp/ouroboros","repo_kind":"official","path":"ouroboros/models/modeling_llama.py","file_url":"https://github.com/thunlp/ouroboros/blob/HEAD/ouroboros/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"3e877f54ec1313b0","entry":"_debug_show_kvcache","repo":"thunlp/Ouroboros","repo_kind":"official","path":"ouroboros/ouroboros.py","file_url":"https://github.com/thunlp/Ouroboros/blob/HEAD/ouroboros/ouroboros.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e877f54ec1313b0"}},{"code_sha256_prefix":"0da5cc8e940c9ab5","entry":"j_make_causal_mask_multilevel","repo":"thunlp/ouroboros","repo_kind":"official","path":"ouroboros/models/mask_making_llama.py","file_url":"https://github.com/thunlp/ouroboros/blob/HEAD/ouroboros/models/mask_making_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0da5cc8e940c9ab5"}},{"code_sha256_prefix":"7940da8888838790","entry":"j_make_causal_mask_with_guess","repo":"thunlp/ouroboros","repo_kind":"official","path":"ouroboros/models/mask_making_llama.py","file_url":"https://github.com/thunlp/ouroboros/blob/HEAD/ouroboros/models/mask_making_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7940da8888838790"}},{"code_sha256_prefix":"07ad96b45bb5eb75","entry":"ouroboros","repo":"thunlp/Ouroboros","repo_kind":"official","path":"ouroboros/ouroboros.py","file_url":"https://github.com/thunlp/Ouroboros/blob/HEAD/ouroboros/ouroboros.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"07ad96b45bb5eb75"}},{"code_sha256_prefix":"c27ae46e25d93b32","entry":"ouroboros_specbench","repo":"thunlp/ouroboros","repo_kind":"official","path":"benchmark/test_specbench/model/ouroboros/ouroboros.py","file_url":"https://github.com/thunlp/ouroboros/blob/HEAD/benchmark/test_specbench/model/ouroboros/ouroboros.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c27ae46e25d93b32"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}