{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/draft-based-approximate-inference-for-llms","title":"Draft-based Approximate Inference for LLMs","arxiv_id":"2506.08373","date":"2025-06-10","proceeding":null,"authors":["Kevin Galim","Ethan Ewer","Wonjun Kang","Minjae Lee","Hyung Il Koo","Kangwook Lee"],"abstract":"Optimizing inference for long-context Large Language Models (LLMs) is increasingly important due to the quadratic compute and linear memory complexity of Transformers. Existing approximation methods, such as key-value (KV) cache dropping, sparse attention, and prompt compression, typically rely on rough predictions of token or KV pair importance. We propose a novel framework for approximate LLM inference that leverages small draft models to more accurately predict the importance of tokens and KV pairs. Specifically, we introduce two instantiations of our proposed framework: (i) SpecKV, which leverages a draft output to accurately assess the importance of each KV pair for more effective KV cache dropping, and (ii) SpecPC, which uses the draft model's attention activations to identify and discard unimportant prompt tokens. To the best of our knowledge, this is the first work to use draft models for approximate LLM inference acceleration, extending their utility beyond traditional lossless speculative decoding. We motivate our methods with theoretical and empirical analyses, and show a strong correlation between the attention patterns of draft and target models. Extensive experiments on long-context benchmarks show that our methods consistently achieve higher accuracy than existing baselines, while preserving the same improvements in memory usage, latency, and throughput. Our code is available at https://github.com/furiosa-ai/draft-based-approx-llm.","url_abs":"https://arxiv.org/abs/2506.08373v1","url_pdf":"https://arxiv.org/pdf/2506.08373v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"draft-based-approximate-inference-for-llms","repo_url":"https://github.com/furiosa-ai/draft-based-approx-llm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.08373","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.08373"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/furiosa-ai/draft-based-approx-llm","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":3,"ran_draft_wrong":2,"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"3ddd62fbad41e698","entry":"build_chat","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"3ddd62fbad41e698"}},{"code_sha256_prefix":"a06d17c88eb31b3d","entry":"build_chat_llama3","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a06d17c88eb31b3d"}},{"code_sha256_prefix":"3597e5670a759378","entry":"key_pruner_query_driven","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"3597e5670a759378"}},{"code_sha256_prefix":"6211e835b472f7ad","entry":"maybe_repeat_kv_before_cache","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"6211e835b472f7ad"}},{"code_sha256_prefix":"d40a43c3d846d891","entry":"compress_kv","repo":"furiosa-ai/draft-based-approx-llm","repo_kind":"official","path":"draft_approx_llm/speckv/util.py","file_url":"https://github.com/furiosa-ai/draft-based-approx-llm/blob/HEAD/draft_approx_llm/speckv/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d40a43c3d846d891"}},{"code_sha256_prefix":"2b96d4ca67c65e40","entry":"generate_aggregated_attention","repo":"furiosa-ai/draft-based-approx-llm","repo_kind":"official","path":"draft_approx_llm/specpc/specpc_utils.py","file_url":"https://github.com/furiosa-ai/draft-based-approx-llm/blob/HEAD/draft_approx_llm/specpc/specpc_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b96d4ca67c65e40"}},{"code_sha256_prefix":"edb10589c3297b21","entry":"patch_model","repo":"furiosa-ai/draft-based-approx-llm","repo_kind":"official","path":"draft_approx_llm/draft_approx_llm.py","file_url":"https://github.com/furiosa-ai/draft-based-approx-llm/blob/HEAD/draft_approx_llm/draft_approx_llm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"edb10589c3297b21"}},{"code_sha256_prefix":"864ba42e7bb2bd2e","entry":"reset_llama_model","repo":"furiosa-ai/draft-based-approx-llm","repo_kind":"official","path":"draft_approx_llm/speckv/llama_util.py","file_url":"https://github.com/furiosa-ai/draft-based-approx-llm/blob/HEAD/draft_approx_llm/speckv/llama_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"864ba42e7bb2bd2e"}},{"code_sha256_prefix":"b547ebd6d3a3089d","entry":"update_llama_model_for_speckv","repo":"furiosa-ai/draft-based-approx-llm","repo_kind":"official","path":"draft_approx_llm/speckv/llama_util.py","file_url":"https://github.com/furiosa-ai/draft-based-approx-llm/blob/HEAD/draft_approx_llm/speckv/llama_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b547ebd6d3a3089d"}},{"code_sha256_prefix":"3dfaf65cb0227004","entry":"vertical_slash_sparse_attention_forward","repo":"furiosa-ai/draft-based-approx-llm","repo_kind":"official","path":"draft_approx_llm/speckv/util.py","file_url":"https://github.com/furiosa-ai/draft-based-approx-llm/blob/HEAD/draft_approx_llm/speckv/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3dfaf65cb0227004"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}