{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gumiho-a-hybrid-architecture-to-prioritize","title":"Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding","arxiv_id":"2503.10135","date":"2025-03-13","proceeding":null,"authors":["Jinze Li","Yixing Xu","Haiduo Huang","Xuanwu Yin","Dong Li","Edith C. H. Ngai","Emad Barsoum"],"abstract":"Speculative decoding (SPD) aims to accelerate the auto-regressive token generation process of a target Large Language Model (LLM). Some approaches employ a draft model with multiple heads to predict a sequence of future tokens, where each head handles a token in the sequence. The target LLM verifies the predicted sequence and accepts aligned tokens, enabling efficient multi-token generation. However, existing methods assume that all tokens within a sequence are equally important, employing identical head structures and relying on a single-generation paradigm, either serial or parallel. To this end, we theoretically demonstrate that initial tokens in the draft sequence are more important than later ones. Building on this insight, we propose Gumiho, a hybrid model combining serial and parallel heads. Specifically, given the critical importance of early tokens, we employ a sophisticated Transformer architecture for the early draft heads in a serial configuration to improve accuracy. For later tokens, we utilize multiple lightweight MLP heads operating in parallel to enhance efficiency. By allocating more advanced model structures and longer running times to the early heads, Gumiho achieves improved overall performance. The experimental results demonstrate that our method outperforms existing approaches, fully validating its effectiveness.","url_abs":"https://arxiv.org/abs/2503.10135v1","url_pdf":"https://arxiv.org/pdf/2503.10135v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.10135","atlas_url":"https://app.syntology.ai/?focus=2503.10135","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.10135"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/AMD-AIG-AIMA/Gumiho","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran":8,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":15,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/cnets.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/cnets.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"9a65a30d006fc96e","entry":"apply_rotary_pos_emb","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/cnets.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/cnets.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a65a30d006fc96e"}},{"code_sha256_prefix":"3519ee6c91b32335","entry":"generate_tree_buffers","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/utils_c.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/utils_c.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3519ee6c91b32335"}},{"code_sha256_prefix":"d27b9a7a10267a63","entry":"generate_tree_buffers","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/utils.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d27b9a7a10267a63"}},{"code_sha256_prefix":"aadc9c83d7ec2701","entry":"initialize_past_key_values","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/kv_cache.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/kv_cache.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aadc9c83d7ec2701"}},{"code_sha256_prefix":"a40c43d3dbc369d3","entry":"pad_path","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/utils.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a40c43d3dbc369d3"}},{"code_sha256_prefix":"dfb6f2c070ac64a8","entry":"pad_path","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/utils_c.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/utils_c.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dfb6f2c070ac64a8"}},{"code_sha256_prefix":"78ae913ae6243c38","entry":"prepare_logits_processor","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/utils.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"78ae913ae6243c38"}},{"code_sha256_prefix":"f6e0c0fc3868b787","entry":"repeat_kv","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/modeling_llama_kv.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/modeling_llama_kv.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6e0c0fc3868b787"}},{"code_sha256_prefix":"272b0f3a92684ed4","entry":"reset_past_key_values","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/utils_c.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/utils_c.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"272b0f3a92684ed4"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/cnets.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/cnets.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"b71b0ed70aa2939c","entry":"rotate_half","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/modeling_llama_kv.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/modeling_llama_kv.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b71b0ed70aa2939c"}},{"code_sha256_prefix":"bad260f6d71db00c","entry":"apply_rotary_pos_emb","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/model/modeling_llama_kv.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/model/modeling_llama_kv.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bad260f6d71db00c"}},{"code_sha256_prefix":"7b6815daba467d5f","entry":"gumiho_forward","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/evaluation/gen_baseline_answer_llama3chat.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/evaluation/gen_baseline_answer_llama3chat.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b6815daba467d5f"}},{"code_sha256_prefix":"8213b0e73fb144d8","entry":"gumiho_forward","repo":"AMD-AIG-AIMA/Gumiho","repo_kind":"found_in_text","path":"gumiho/evaluation/gen_baseline_answer_vicuna.py","file_url":"https://github.com/AMD-AIG-AIMA/Gumiho/blob/HEAD/gumiho/evaluation/gen_baseline_answer_vicuna.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8213b0e73fb144d8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}