{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/magicdec-breaking-the-latency-throughput","title":"MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding","arxiv_id":"2408.11049","date":"2024-08-20","proceeding":null,"authors":["Ranajoy Sadhukhan","Jian Chen","Zhuoming Chen","Vashisth Tiwari","Ruihang Lai","Jinyuan Shi","Ian En-Hsu Yen","Avner May","Tianqi Chen","Beidi Chen"],"abstract":"Large Language Models (LLMs) have become more prevalent in long-context applications such as interactive chatbots, document analysis, and agent workflows, but it is challenging to serve long-context requests with low latency and high throughput. Speculative decoding (SD) is a widely used technique to reduce latency losslessly, but the conventional wisdom suggests that its efficacy is limited to small batch sizes. In MagicDec, we show that surprisingly SD can achieve speedup even for a high throughput inference regime for moderate to long sequences. More interestingly, an intelligent drafting strategy can achieve better speedup with increasing batch size based on our rigorous analysis. MagicDec first identifies the bottleneck shifts with increasing batch size and sequence length, and uses these insights to deploy SD more effectively for high throughput inference. We leverage draft model with sparse KV cache to address the KV bottleneck, which scales with both sequence length and batch size. Additionally, we propose a theoretical model to select the optimal drafting strategy for maximum speedup. Our work highlights the broad applicability of speculative decoding in long-context serving, as it can enhance throughput and reduce latency without compromising accuracy. For moderate to long sequences, we demonstrate up to 2.51x speedup for Llama3.1-8B when serving batch sizes ranging from 32 to 256 on various types of hardware and tasks.","url_abs":"https://arxiv.org/abs/2408.11049v4","url_pdf":"https://arxiv.org/pdf/2408.11049v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"magicdec-breaking-the-latency-throughput","repo_url":"https://github.com/infini-ai-lab/magicdec","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.11049","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.11049"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/infini-ai-lab/magicdec","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran_honours":1,"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"infini-ai-lab/magicdec","repo_kind":"official","path":"Engine/utils.py","file_url":"https://github.com/infini-ai-lab/magicdec/blob/HEAD/Engine/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"aba86ffbed27ba65","entry":"find_multiple","repo":"infini-ai-lab/magicdec","repo_kind":"official","path":"Engine/SnapKV/model.py","file_url":"https://github.com/infini-ai-lab/magicdec/blob/HEAD/Engine/SnapKV/model.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aba86ffbed27ba65"}},{"code_sha256_prefix":"ed50d08882ec36a7","entry":"unrepeat_kv","repo":"infini-ai-lab/magicdec","repo_kind":"official","path":"Engine/utils.py","file_url":"https://github.com/infini-ai-lab/magicdec/blob/HEAD/Engine/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed50d08882ec36a7"}},{"code_sha256_prefix":"55ec8fea60e1d7e7","entry":"dynamically_quantize_per_channel","repo":"infini-ai-lab/magicdec","repo_kind":"official","path":"Engine/quantize.py","file_url":"https://github.com/infini-ai-lab/magicdec/blob/HEAD/Engine/quantize.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55ec8fea60e1d7e7"}},{"code_sha256_prefix":"e4f851f13f6f224e","entry":"init_dist","repo":"infini-ai-lab/magicdec","repo_kind":"official","path":"Engine/tp.py","file_url":"https://github.com/infini-ai-lab/magicdec/blob/HEAD/Engine/tp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e4f851f13f6f224e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}