{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hydra-sequentially-dependent-draft-heads-for","title":"Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding","arxiv_id":"2402.05109","date":"2024-02-07","proceeding":null,"authors":["Zachary Ankner","Rishab Parthasarathy","Aniruddha Nrusimha","Christopher Rinard","Jonathan Ragan-Kelley","William Brandon"],"abstract":"To combat the memory bandwidth-bound nature of autoregressive LLM inference, previous research has proposed the speculative decoding frame-work. To perform speculative decoding, a small draft model proposes candidate continuations of the input sequence that are then verified in parallel by the base model. One way to specify the draft model, as used in the recent Medusa decoding framework, is as a collection of lightweight heads, called draft heads, that operate on the base model's hidden states. To date, all existing draft heads have been sequentially independent, meaning that they speculate tokens in the candidate continuation independently of any preceding tokens in the candidate continuation. In this work, we propose Hydra heads: a sequentially-dependent drop-in replacement for standard draft heads that significantly improves the accuracy of draft head speculation. We further explore the design space of Hydra head training objectives and architectures, and propose a carefully tuned Hydra head recipe, which we call Hydra++, that improves decoding throughput by up to 1.31x and 2.70x compared to Medusa decoding and autoregressive de-coding respectively. Overall, Hydra heads are a simple and well-motivated intervention on standard draft heads that significantly improve the end-to-end speed of draft head-based speculative decoding. We make our code publicly available at https://github.com/zankner/Hydra.","url_abs":"https://arxiv.org/abs/2402.05109v2","url_pdf":"https://arxiv.org/pdf/2402.05109v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hydra-sequentially-dependent-draft-heads-for","repo_url":"https://github.com/zankner/hydra","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"hydra","method_name":"Hydra"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.05109","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.05109"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zankner/hydra","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"395871bd98d29b08","entry":"generate_hydra_buffers","repo":"zankner/hydra","repo_kind":"official","path":"hydra/model/utils.py","file_url":"https://github.com/zankner/hydra/blob/HEAD/hydra/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"395871bd98d29b08"}},{"code_sha256_prefix":"813156115eaf61cc","entry":"get_constant_schedule_with_warmup","repo":"zankner/hydra","repo_kind":"official","path":"hydra/train/utils.py","file_url":"https://github.com/zankner/hydra/blob/HEAD/hydra/train/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"813156115eaf61cc"}},{"code_sha256_prefix":"48e7f6a3de154240","entry":"get_cosine_schedule_with_warmup","repo":"zankner/hydra","repo_kind":"official","path":"hydra/train/utils.py","file_url":"https://github.com/zankner/hydra/blob/HEAD/hydra/train/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"48e7f6a3de154240"}},{"code_sha256_prefix":"b0d392034f20d785","entry":"get_linear_schedule_with_warmup","repo":"zankner/hydra","repo_kind":"official","path":"hydra/train/utils.py","file_url":"https://github.com/zankner/hydra/blob/HEAD/hydra/train/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0d392034f20d785"}},{"code_sha256_prefix":"d2d232761fe51ccb","entry":"initialize_past_key_values","repo":"zankner/hydra","repo_kind":"official","path":"hydra/model/kv_cache.py","file_url":"https://github.com/zankner/hydra/blob/HEAD/hydra/model/kv_cache.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d2d232761fe51ccb"}},{"code_sha256_prefix":"a0751eee3897a746","entry":"pad_path","repo":"zankner/hydra","repo_kind":"official","path":"hydra/model/utils.py","file_url":"https://github.com/zankner/hydra/blob/HEAD/hydra/model/utils.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a0751eee3897a746"}},{"code_sha256_prefix":"f6e0c0fc3868b787","entry":"repeat_kv","repo":"zankner/hydra","repo_kind":"official","path":"hydra/model/modeling_llama_kv.py","file_url":"https://github.com/zankner/hydra/blob/HEAD/hydra/model/modeling_llama_kv.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6e0c0fc3868b787"}},{"code_sha256_prefix":"b71b0ed70aa2939c","entry":"rotate_half","repo":"zankner/hydra","repo_kind":"official","path":"hydra/model/modeling_llama_kv.py","file_url":"https://github.com/zankner/hydra/blob/HEAD/hydra/model/modeling_llama_kv.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b71b0ed70aa2939c"}},{"code_sha256_prefix":"bad260f6d71db00c","entry":"apply_rotary_pos_emb","repo":"zankner/hydra","repo_kind":"official","path":"hydra/model/modeling_llama_kv.py","file_url":"https://github.com/zankner/hydra/blob/HEAD/hydra/model/modeling_llama_kv.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bad260f6d71db00c"}},{"code_sha256_prefix":"2475708ca93552ac","entry":"initialize_hydra","repo":"zankner/hydra","repo_kind":"official","path":"hydra/model/utils.py","file_url":"https://github.com/zankner/hydra/blob/HEAD/hydra/model/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2475708ca93552ac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}