{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/triforce-lossless-acceleration-of-long","title":"TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding","arxiv_id":"2404.11912","date":"2024-04-18","proceeding":null,"authors":["Hanshi Sun","Zhuoming Chen","Xinyu Yang","Yuandong Tian","Beidi Chen"],"abstract":"With large language models (LLMs) widely deployed in long content generation recently, there has emerged an increasing demand for efficient long-sequence inference support. However, key-value (KV) cache, which is stored to avoid re-computation, has emerged as a critical bottleneck by growing linearly in size with the sequence length. Due to the auto-regressive nature of LLMs, the entire KV cache will be loaded for every generated token, resulting in low utilization of computational cores and high latency. While various compression methods for KV cache have been proposed to alleviate this issue, they suffer from degradation in generation quality. We introduce TriForce, a hierarchical speculative decoding system that is scalable for long sequence generation. This approach leverages the original model weights and dynamic sparse KV cache via retrieval as a draft model, which serves as an intermediate layer in the hierarchy and is further speculated by a smaller model to reduce its drafting latency. TriForce not only facilitates impressive speedups for Llama2-7B-128K, achieving up to 2.31$\\times$ on an A100 GPU but also showcases scalability in handling even longer contexts. For the offloading setting on two RTX 4090 GPUs, TriForce achieves 0.108s/token$\\unicode{x2014}$only half as slow as the auto-regressive baseline on an A100, which attains 7.78$\\times$ on our optimized offloading system. Additionally, TriForce performs 4.86$\\times$ than DeepSpeed-Zero-Inference on a single RTX 4090 GPU. TriForce's robustness is highlighted by its consistently outstanding performance across various temperatures. The code is available at https://github.com/Infini-AI-Lab/TriForce.","url_abs":"https://arxiv.org/abs/2404.11912v3","url_pdf":"https://arxiv.org/pdf/2404.11912v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"triforce-lossless-acceleration-of-long","repo_url":"https://github.com/Infini-AI-Lab/TriForce","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.11912","atlas_url":"https://app.syntology.ai/?focus=2404.11912","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.11912"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Infini-AI-Lab/TriForce","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"ran":2,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"Infini-AI-Lab/TriForce","repo_kind":"official","path":"models/tensor_op.py","file_url":"https://github.com/Infini-AI-Lab/TriForce/blob/HEAD/models/tensor_op.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"0fe82a947dc39b42","entry":"apply_rotary_pos_emb","repo":"Infini-AI-Lab/TriForce","repo_kind":"official","path":"models/tensor_op.py","file_url":"https://github.com/Infini-AI-Lab/TriForce/blob/HEAD/models/tensor_op.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fe82a947dc39b42"}},{"code_sha256_prefix":"f101335059827a99","entry":"apply_rotary_pos_emb_single","repo":"Infini-AI-Lab/TriForce","repo_kind":"official","path":"models/modeling_llama_68m.py","file_url":"https://github.com/Infini-AI-Lab/TriForce/blob/HEAD/models/modeling_llama_68m.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f101335059827a99"}},{"code_sha256_prefix":"e4dedff82f675b83","entry":"get_sampling_logits","repo":"Infini-AI-Lab/TriForce","repo_kind":"official","path":"utils/SpecTree_TP.py","file_url":"https://github.com/Infini-AI-Lab/TriForce/blob/HEAD/utils/SpecTree_TP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4dedff82f675b83"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"Infini-AI-Lab/TriForce","repo_kind":"official","path":"models/modeling_llama_68m.py","file_url":"https://github.com/Infini-AI-Lab/TriForce/blob/HEAD/models/modeling_llama_68m.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"c4404a21ff202740","entry":"draft_run_capture_graph","repo":"Infini-AI-Lab/TriForce","repo_kind":"official","path":"utils/graph_infer.py","file_url":"https://github.com/Infini-AI-Lab/TriForce/blob/HEAD/utils/graph_infer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4404a21ff202740"}},{"code_sha256_prefix":"aed1de4698a72f4f","entry":"model_verify_capture_graph","repo":"Infini-AI-Lab/TriForce","repo_kind":"official","path":"utils/graph_infer.py","file_url":"https://github.com/Infini-AI-Lab/TriForce/blob/HEAD/utils/graph_infer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aed1de4698a72f4f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}