{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/longspec-long-context-speculative-decoding","title":"LongSpec: Long-Context Speculative Decoding with Efficient Drafting and Verification","arxiv_id":"2502.17421","date":"2025-02-24","proceeding":null,"authors":["Penghui Yang","Cunxiao Du","Fengzhuo Zhang","Haonan Wang","Tianyu Pang","Chao Du","Bo An"],"abstract":"Speculative decoding has become a promising technique to mitigate the high inference latency of autoregressive decoding in Large Language Models (LLMs). Despite its promise, the effective application of speculative decoding in LLMs still confronts three key challenges: the increasing memory demands of the draft model, the distribution shift between the short-training corpora and long-context inference, and inefficiencies in attention implementation. In this work, we enhance the performance of speculative decoding in long-context settings by addressing these challenges. First, we propose a memory-efficient draft model with a constant-sized Key-Value (KV) cache. Second, we introduce novel position indices for short-training data, enabling seamless adaptation from short-context training to long-context inference. Finally, we present an innovative attention aggregation method that combines fast implementations for prefix computation with standard attention for tree mask handling, effectively resolving the latency and memory inefficiencies of tree decoding. Our approach achieves strong results on various long-context tasks, including repository-level code completion, long-context summarization, and o1-like long reasoning tasks, demonstrating significant improvements in latency reduction. The code is available at https://github.com/sail-sg/LongSpec.","url_abs":"https://arxiv.org/abs/2502.17421v1","url_pdf":"https://arxiv.org/pdf/2502.17421v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"longspec-long-context-speculative-decoding","repo_url":"https://github.com/sail-sg/longspec","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"code-completion","task_name":"Code Completion"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.17421","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.17421"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sail-sg/longspec","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"unverified":8},"by_repo_kind":{"official":{"samples":10,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bac65c3dafaec040","entry":"apply_rotary_pos_emb","repo":"sail-sg/longspec","repo_kind":"official","path":"longspec/train/models/qwen2.py","file_url":"https://github.com/sail-sg/longspec/blob/HEAD/longspec/train/models/qwen2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bac65c3dafaec040"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"sail-sg/longspec","repo_kind":"official","path":"longspec/train/models/qwen2.py","file_url":"https://github.com/sail-sg/longspec/blob/HEAD/longspec/train/models/qwen2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"8d0a17385afef041","entry":"build_dataloader","repo":"sail-sg/longspec","repo_kind":"official","path":"longspec/train/general_util/evaluator.py","file_url":"https://github.com/sail-sg/longspec/blob/HEAD/longspec/train/general_util/evaluator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d0a17385afef041"}},{"code_sha256_prefix":"a8d00b249de11718","entry":"enable_gradient_checkpointing","repo":"sail-sg/longspec","repo_kind":"official","path":"longspec/train/models/utils.py","file_url":"https://github.com/sail-sg/longspec/blob/HEAD/longspec/train/models/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8d00b249de11718"}},{"code_sha256_prefix":"bb5b320c91cd4e2d","entry":"extract_and_rename","repo":"sail-sg/longspec","repo_kind":"official","path":"longspec/train/trainer_base_ds_mul_fs_tp.py","file_url":"https://github.com/sail-sg/longspec/blob/HEAD/longspec/train/trainer_base_ds_mul_fs_tp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb5b320c91cd4e2d"}},{"code_sha256_prefix":"59492727dafc7ec5","entry":"forward_step","repo":"sail-sg/longspec","repo_kind":"official","path":"longspec/train/trainer_base_ds_mul_fs_tp.py","file_url":"https://github.com/sail-sg/longspec/blob/HEAD/longspec/train/trainer_base_ds_mul_fs_tp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59492727dafc7ec5"}},{"code_sha256_prefix":"c4393aecafc8b841","entry":"get_zero_stage","repo":"sail-sg/longspec","repo_kind":"official","path":"longspec/train/trainer_base_ds_mul_fs_tp.py","file_url":"https://github.com/sail-sg/longspec/blob/HEAD/longspec/train/trainer_base_ds_mul_fs_tp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4393aecafc8b841"}},{"code_sha256_prefix":"58e6542897e7d424","entry":"maybe_contiguous","repo":"sail-sg/longspec","repo_kind":"official","path":"longspec/train/models/triton_tree_attn.py","file_url":"https://github.com/sail-sg/longspec/blob/HEAD/longspec/train/models/triton_tree_attn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58e6542897e7d424"}},{"code_sha256_prefix":"d0340dda6fca386f","entry":"rounded_multiple","repo":"sail-sg/longspec","repo_kind":"official","path":"longspec/train/models/triton_tree_attn.py","file_url":"https://github.com/sail-sg/longspec/blob/HEAD/longspec/train/models/triton_tree_attn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0340dda6fca386f"}},{"code_sha256_prefix":"a8073503b1d41587","entry":"torch_tree_attention","repo":"sail-sg/longspec","repo_kind":"official","path":"longspec/train/models/qwen2.py","file_url":"https://github.com/sail-sg/longspec/blob/HEAD/longspec/train/models/qwen2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8073503b1d41587"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}