{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/speculative-prefill-turbocharging-ttft-with","title":"Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation","arxiv_id":"2502.02789","date":"2025-02-05","proceeding":null,"authors":["Jingyu Liu","Beidi Chen","Ce Zhang"],"abstract":"Improving time-to-first-token (TTFT) is an essentially important objective in modern large language model (LLM) inference engines. Optimizing TTFT directly results in higher maximal QPS and meets the requirements of many critical applications. However, boosting TTFT is notoriously challenging since it is compute-bounded and the performance bottleneck shifts from the self-attention that many prior works focus on to the MLP part. In this work, we present SpecPrefill, a training free framework that accelerates the inference TTFT for both long and medium context queries based on the following insight: LLMs are generalized enough to preserve the quality given only a carefully chosen subset of prompt tokens. At its core, SpecPrefill leverages a lightweight model to speculate locally important tokens based on the context. These tokens, along with the necessary positional information, are then sent to the main model for processing. We evaluate SpecPrefill with a diverse set of tasks, followed by a comprehensive benchmarking of performance improvement both in a real end-to-end setting and ablation studies. SpecPrefill manages to serve Llama-3.1-405B-Instruct-FP8 with up to 7$\\times$ maximal end-to-end QPS on real downstream tasks and 7.66$\\times$ TTFT improvement.","url_abs":"https://arxiv.org/abs/2502.02789v2","url_pdf":"https://arxiv.org/pdf/2502.02789v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"speculative-prefill-turbocharging-ttft-with","repo_url":"https://github.com/Jingyu6/speculative_prefill","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2502.02789","atlas_url":"https://app.syntology.ai/?focus=2502.02789","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.02789"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/openai/openai-python","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Jingyu6/speculative_prefill","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cbf40a92b65b6d6f","entry":"send_query","repo":"Jingyu6/speculative_prefill","repo_kind":"official","path":"eval/qps_client.py","file_url":"https://github.com/Jingyu6/speculative_prefill/blob/HEAD/eval/qps_client.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbf40a92b65b6d6f"}},{"code_sha256_prefix":"751ed5bbbe3108f7","entry":"split_num_cache_blocks_evenly","repo":"Jingyu6/speculative_prefill","repo_kind":"official","path":"speculative_prefill/vllm_patch/worker/spec_prefill_worker.py","file_url":"https://github.com/Jingyu6/speculative_prefill/blob/HEAD/speculative_prefill/vllm_patch/worker/spec_prefill_worker.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"751ed5bbbe3108f7"}},{"code_sha256_prefix":"ec827626111b4b80","entry":"prepare_datasets","repo":"Jingyu6/speculative_prefill","repo_kind":"official","path":"eval/qps_client.py","file_url":"https://github.com/Jingyu6/speculative_prefill/blob/HEAD/eval/qps_client.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec827626111b4b80"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}