{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hydragen-high-throughput-llm-inference-with","title":"Hydragen: High-Throughput LLM Inference with Shared Prefixes","arxiv_id":"2402.05099","date":"2024-02-07","proceeding":null,"authors":["Jordan Juravsky","Bradley Brown","Ryan Ehrlich","Daniel Y. Fu","Christopher Ré","Azalia Mirhoseini"],"abstract":"Transformer-based large language models (LLMs) are now deployed to hundreds of millions of users. LLM inference is commonly performed on batches of sequences that share a prefix, such as few-shot examples or a chatbot system prompt. Decoding in this large-batch setting can be bottlenecked by the attention operation, which reads large key-value (KV) caches from memory and computes inefficient matrix-vector products for every sequence in the batch. In this work, we introduce Hydragen, a hardware-aware exact implementation of attention with shared prefixes. Hydragen computes attention over the shared prefix and unique suffixes separately. This decomposition enables efficient prefix attention by batching queries together across sequences, reducing redundant memory reads and enabling the use of hardware-friendly matrix multiplications. Our method can improve end-to-end CodeLlama-13b throughput by up to 32x against competitive baselines, with speedup growing with the batch size and shared prefix length. Hydragen also enables the use of very long shared contexts: with a large batch size, increasing the prefix length from 1K to 16K tokens decreases Hydragen throughput by less than 15%, while the throughput of baselines drops by over 90%. Hydragen generalizes beyond simple prefix-suffix decomposition and can be applied to tree-based prompt sharing patterns, allowing us to further reduce inference time on competitive programming problems by 55%.","url_abs":"https://arxiv.org/abs/2402.05099v2","url_pdf":"https://arxiv.org/pdf/2402.05099v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hydragen-high-throughput-llm-inference-with","repo_url":"https://github.com/jordan-benjamin/hydragen","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"16k","task_name":"16k"},{"task_slug":"chatbot","task_name":"Chatbot"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.05099","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.05099"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jordan-benjamin/hydragen","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8d800826de11142a","entry":"combine_lse_torch","repo":"jordan-benjamin/hydragen","repo_kind":"official","path":"hydragen/attention.py","file_url":"https://github.com/jordan-benjamin/hydragen/blob/HEAD/hydragen/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d800826de11142a"}},{"code_sha256_prefix":"0479fb5f08e12e14","entry":"mean","repo":"jordan-benjamin/hydragen","repo_kind":"official","path":"hydragen/utils.py","file_url":"https://github.com/jordan-benjamin/hydragen/blob/HEAD/hydragen/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0479fb5f08e12e14"}},{"code_sha256_prefix":"09b828d64f672f80","entry":"pick_split_k","repo":"jordan-benjamin/hydragen","repo_kind":"official","path":"hydragen/flash.py","file_url":"https://github.com/jordan-benjamin/hydragen/blob/HEAD/hydragen/flash.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"09b828d64f672f80"}},{"code_sha256_prefix":"42a76a748d975a17","entry":"rdiff","repo":"jordan-benjamin/hydragen","repo_kind":"official","path":"hydragen/utils.py","file_url":"https://github.com/jordan-benjamin/hydragen/blob/HEAD/hydragen/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a76a748d975a17"}},{"code_sha256_prefix":"3f29ee6d9371c9c8","entry":"repeat_to_batch_size","repo":"jordan-benjamin/hydragen","repo_kind":"official","path":"hydragen/llama.py","file_url":"https://github.com/jordan-benjamin/hydragen/blob/HEAD/hydragen/llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3f29ee6d9371c9c8"}},{"code_sha256_prefix":"39bc424e73a7ab48","entry":"variance","repo":"jordan-benjamin/hydragen","repo_kind":"official","path":"hydragen/utils.py","file_url":"https://github.com/jordan-benjamin/hydragen/blob/HEAD/hydragen/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"39bc424e73a7ab48"}},{"code_sha256_prefix":"245f6337807e900c","entry":"make_needle_haystack","repo":"jordan-benjamin/hydragen","repo_kind":"official","path":"hydragen/haystack.py","file_url":"https://github.com/jordan-benjamin/hydragen/blob/HEAD/hydragen/haystack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"245f6337807e900c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}