{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decoding-speculative-decoding","title":"Decoding Speculative Decoding","arxiv_id":"2402.01528","date":"2024-02-02","proceeding":null,"authors":["Minghao Yan","Saurabh Agarwal","Shivaram Venkataraman"],"abstract":"Speculative Decoding is a widely used technique to speed up inference for Large Language Models (LLMs) without sacrificing quality. When performing inference, speculative decoding uses a smaller draft model to generate speculative tokens and then uses the target LLM to verify those draft tokens. The speedup provided by speculative decoding heavily depends on the choice of the draft model. In this work, we perform a detailed study comprising over 350 experiments with LLaMA-65B and OPT-66B using speculative decoding and delineate the factors that affect the performance gain provided by speculative decoding. Our experiments indicate that the performance of speculative decoding depends heavily on the latency of the draft model, and the draft model's capability in language modeling does not correlate strongly with its performance in speculative decoding. Based on these insights we explore a new design space for draft models and design hardware-efficient draft models for speculative decoding. Our newly designed draft model for LLaMA-65B can provide 111% higher throughput than existing draft models and can generalize further to the LLaMA-2 model family and supervised fine-tuned models.","url_abs":"https://arxiv.org/abs/2402.01528v3","url_pdf":"https://arxiv.org/pdf/2402.01528v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decoding-speculative-decoding","repo_url":"https://github.com/uw-mad-dash/decoding-speculative-decoding","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.01528","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.01528"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uw-mad-dash/decoding-speculative-decoding","reach":null}],"summary":{"ran_draft_wrong":2,"ran_fixture":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"7d0dfe822f8d993a","entry":"json_loader","repo":"uw-mad-dash/decoding-speculative-decoding","repo_kind":"official","path":"spec_decoding_demo.py","file_url":"https://github.com/uw-mad-dash/decoding-speculative-decoding/blob/HEAD/spec_decoding_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d0dfe822f8d993a"}},{"code_sha256_prefix":"cdde66fff666931e","entry":"oracle_verification","repo":"uw-mad-dash/decoding-speculative-decoding","repo_kind":"official","path":"spec_decoding_deployment.py","file_url":"https://github.com/uw-mad-dash/decoding-speculative-decoding/blob/HEAD/spec_decoding_deployment.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cdde66fff666931e"}},{"code_sha256_prefix":"979d0e45be740936","entry":"parse_tensors_from_file","repo":"uw-mad-dash/decoding-speculative-decoding","repo_kind":"official","path":"spec_decoding_demo.py","file_url":"https://github.com/uw-mad-dash/decoding-speculative-decoding/blob/HEAD/spec_decoding_demo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"979d0e45be740936"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}