{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mixture-of-attentions-for-speculative","title":"Mixture of Attentions For Speculative Decoding","arxiv_id":"2410.03804","date":"2024-10-04","proceeding":null,"authors":["Matthieu Zimmer","Milan Gritta","Gerasimos Lampouras","Haitham Bou Ammar","Jun Wang"],"abstract":"The growth in the number of parameters of Large Language Models (LLMs) has led to a significant surge in computational requirements, making them challenging and costly to deploy. Speculative decoding (SD) leverages smaller models to efficiently propose future tokens, which are then verified by the LLM in parallel. Small models that utilise activations from the LLM currently achieve the fastest decoding speeds. However, we identify several limitations of SD models including the lack of on-policyness during training and partial observability. To address these shortcomings, we propose a more grounded architecture for small models by introducing a Mixture of Attentions for SD. Our novel architecture can be applied in two scenarios: a conventional single device deployment and a novel client-server deployment where the small model is hosted on a consumer device and the LLM on a server. In a single-device scenario, we demonstrate state-of-the-art speedups improving EAGLE-2 by 9.5% and its acceptance length by 25%. In a client-server setting, our experiments demonstrate: 1) state-of-the-art latencies with minimal calls to the server for different network conditions, and 2) in the event of a complete disconnection, our approach can maintain higher accuracy compared to other SD methods and demonstrates advantages over API calls to LLMs, which would otherwise be unable to continue the generation process.","url_abs":"https://arxiv.org/abs/2410.03804v1","url_pdf":"https://arxiv.org/pdf/2410.03804v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mixture-of-attentions-for-speculative","repo_url":"https://github.com/huawei-noah/hebo","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.03804","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.03804"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/huawei-noah/hebo","reach":{"status":"ok"}}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"c264fcfb9e72ee00","entry":"construct_hidden","repo":"huawei-noah/hebo","repo_kind":"official","path":"HEBO/hebo/models/util.py","file_url":"https://github.com/huawei-noah/hebo/blob/HEAD/HEBO/hebo/models/util.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c264fcfb9e72ee00"}},{"code_sha256_prefix":"3df50a3290357381","entry":"filter_nan","repo":"huawei-noah/hebo","repo_kind":"official","path":"HEBO/hebo/models/util.py","file_url":"https://github.com/huawei-noah/hebo/blob/HEAD/HEBO/hebo/models/util.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3df50a3290357381"}},{"code_sha256_prefix":"164d85100f303f3c","entry":"get_init_pop","repo":"huawei-noah/hebo","repo_kind":"official","path":"HEBO/hebo/acq_optimizers/evolution_optimizer.py","file_url":"https://github.com/huawei-noah/hebo/blob/HEAD/HEBO/hebo/acq_optimizers/evolution_optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"164d85100f303f3c"}},{"code_sha256_prefix":"2457b75be8691285","entry":"space_to_pymoo_vars","repo":"huawei-noah/hebo","repo_kind":"official","path":"HEBO/hebo/acq_optimizers/evolution_optimizer.py","file_url":"https://github.com/huawei-noah/hebo/blob/HEAD/HEBO/hebo/acq_optimizers/evolution_optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2457b75be8691285"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}