{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/parallel-speculative-decoding-with-adaptive","title":"Parallel Speculative Decoding with Adaptive Draft Length","arxiv_id":"2408.11850","date":"2024-08-13","proceeding":null,"authors":["Tianyu Liu","Yun Li","Qitan Lv","Kai Liu","Jianchen Zhu","Winston Hu"],"abstract":"Speculative decoding (SD), where an extra draft model is employed to provide multiple \\textit{draft} tokens first and then the original target model verifies these tokens in parallel, has shown great power for LLM inference acceleration. However, existing SD methods suffer from the mutual waiting problem, i.e., the target model gets stuck when the draft model is \\textit{guessing} tokens, and vice versa. This problem is directly incurred by the asynchronous execution of the draft model and the target model, and is exacerbated due to the fixed draft length in speculative decoding. To address these challenges, we propose a conceptually simple, flexible, and general framework to boost speculative decoding, namely \\textbf{P}arallel sp\\textbf{E}culative decoding with \\textbf{A}daptive d\\textbf{R}aft \\textbf{L}ength (PEARL). Specifically, PEARL proposes \\textit{pre-verify} to verify the first draft token in advance during the drafting phase, and \\textit{post-verify} to generate more draft tokens during the verification phase. PEARL parallels the drafting phase and the verification phase via applying the two strategies, and achieves adaptive draft length for different scenarios, which effectively alleviates the mutual waiting problem. Moreover, we theoretically demonstrate that the mean accepted tokens of PEARL is more than existing \\textit{draft-then-verify} works. Experiments on various text generation benchmarks demonstrate the effectiveness of our \\name, leading to a superior speedup performance up to \\textbf{3.79$\\times$} and \\textbf{1.52$\\times$}, compared to auto-regressive decoding and vanilla speculative decoding, respectively.","url_abs":"https://arxiv.org/abs/2408.11850v2","url_pdf":"https://arxiv.org/pdf/2408.11850v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"parallel-speculative-decoding-with-adaptive","repo_url":"https://github.com/smart-lty/parallelspeculativedecoding","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.11850","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.11850"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/smart-lty/ParallelSpeculativeDecoding","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/smart-lty/parallelspeculativedecoding","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran_draft_wrong":4,"ran":2,"ran_fixture":1},"by_repo_kind":{"official":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"288b21dd530b5da3","entry":"find_list_markers","repo":"smart-lty/parallelspeculativedecoding","repo_kind":"official","path":"applications.py","file_url":"https://github.com/smart-lty/parallelspeculativedecoding/blob/HEAD/applications.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"288b21dd530b5da3"}},{"code_sha256_prefix":"ea7422602cf8d39a","entry":"norm_logits","repo":"smart-lty/ParallelSpeculativeDecoding","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/smart-lty/ParallelSpeculativeDecoding/blob/HEAD/src/util.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea7422602cf8d39a"}},{"code_sha256_prefix":"fcb64b361ddd54aa","entry":"read_results","repo":"smart-lty/ParallelSpeculativeDecoding","repo_kind":"official","path":"benchmark/eval_mgsm.py","file_url":"https://github.com/smart-lty/ParallelSpeculativeDecoding/blob/HEAD/benchmark/eval_mgsm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fcb64b361ddd54aa"}},{"code_sha256_prefix":"9c206ca20df50d10","entry":"read_results","repo":"smart-lty/ParallelSpeculativeDecoding","repo_kind":"official","path":"benchmark/eval_mt_bench.py","file_url":"https://github.com/smart-lty/ParallelSpeculativeDecoding/blob/HEAD/benchmark/eval_mt_bench.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9c206ca20df50d10"}},{"code_sha256_prefix":"ffa10e3a85932ece","entry":"sample","repo":"smart-lty/ParallelSpeculativeDecoding","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/smart-lty/ParallelSpeculativeDecoding/blob/HEAD/src/util.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ffa10e3a85932ece"}},{"code_sha256_prefix":"025c3469ca2fb4c3","entry":"top_k_top_p_filter","repo":"smart-lty/ParallelSpeculativeDecoding","repo_kind":"official","path":"src/util.py","file_url":"https://github.com/smart-lty/ParallelSpeculativeDecoding/blob/HEAD/src/util.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"025c3469ca2fb4c3"}},{"code_sha256_prefix":"124313a451530c21","entry":"truncate_list","repo":"smart-lty/parallelspeculativedecoding","repo_kind":"official","path":"applications.py","file_url":"https://github.com/smart-lty/parallelspeculativedecoding/blob/HEAD/applications.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"124313a451530c21"}},{"code_sha256_prefix":"79986b5722842265","entry":"user","repo":"smart-lty/parallelspeculativedecoding","repo_kind":"official","path":"applications.py","file_url":"https://github.com/smart-lty/parallelspeculativedecoding/blob/HEAD/applications.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"79986b5722842265"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}