{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hyena-hierarchy-towards-larger-convolutional","title":"Hyena Hierarchy: Towards Larger Convolutional Language Models","arxiv_id":"2302.10866","date":"2023-02-21","proceeding":null,"authors":["Michael Poli","Stefano Massaroli","Eric Nguyen","Daniel Y. Fu","Tri Dao","Stephen Baccus","Yoshua Bengio","Stefano Ermon","Christopher Ré"],"abstract":"Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the core building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. We set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.","url_abs":"https://arxiv.org/abs/2302.10866v3","url_pdf":"https://arxiv.org/pdf/2302.10866v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hyena-hierarchy-towards-larger-convolutional","repo_url":"https://github.com/hazyresearch/safari","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"hyena-hierarchy-towards-larger-convolutional","repo_url":"https://github.com/Suro-One/Hyena-Hierarchy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"hyena-hierarchy-towards-larger-convolutional","repo_url":"https://github.com/i404788/s5-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MPL-2.0"}},{"paper_slug":"hyena-hierarchy-towards-larger-convolutional","repo_url":"https://github.com/lindermanlab/S5","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"hyena-hierarchy-towards-larger-convolutional","repo_url":"https://github.com/togethercomputer/stripedhyena","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"hyena-hierarchy-towards-larger-convolutional","repo_url":"https://github.com/MindSpore-scientific-2/code-4/tree/main/Hyena-A-Convolutional-Neural-Network-for-Modelling-Sentences","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"hyena-hierarchy-towards-larger-convolutional","repo_url":"https://github.com/expz/annotated-hyena","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"2k","task_name":"2k"},{"task_slug":null,"task_name":"8k"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Hyena-3-slim","rank_in_archive_order":37,"of":89,"metrics":{"Test perplexity":"18.5"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Hyena-3","rank_in_archive_order":39,"of":89,"metrics":{"Test perplexity":"18.6"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-boolq","task":"Question Answering","dataset":"BoolQ","model":"Hyena","rank_in_archive_order":64,"of":65,"metrics":{"Accuracy":"51.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2302.10866","atlas_url":"https://app.syntology.ai/?focus=2302.10866","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.10866"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazyresearch/safari","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lindermanlab/S5","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/i404788/s5-pytorch","reach":{"status":"ok","spdx":"MPL-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific-2/code-4/tree/main/Hyena-A-Convolutional-Neural-Network-for-Modelling-Sentences","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/togethercomputer/stripedhyena","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Suro-One/Hyena-Hierarchy","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/expz/annotated-hyena","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":4,"ran_fixture":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"d3511a9b66800bd1","entry":"build_vocab","repo":"Suro-One/Hyena-Hierarchy","repo_kind":"listed","path":"hyena-split-memory.py","file_url":"https://github.com/Suro-One/Hyena-Hierarchy/blob/HEAD/hyena-split-memory.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3511a9b66800bd1"}},{"code_sha256_prefix":"c75ab6c159dc8368","entry":"expand_model_vocab","repo":"Suro-One/Hyena-Hierarchy","repo_kind":"listed","path":"hyena-split-memory.py","file_url":"https://github.com/Suro-One/Hyena-Hierarchy/blob/HEAD/hyena-split-memory.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c75ab6c159dc8368"}},{"code_sha256_prefix":"6736debb6f8316b9","entry":"fftconv_ref","repo":"hazyresearch/safari","repo_kind":"official","path":"src/models/sequence/hyena.py","file_url":"https://github.com/hazyresearch/safari/blob/HEAD/src/models/sequence/hyena.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6736debb6f8316b9"}},{"code_sha256_prefix":"d3fbe2e426af0f8c","entry":"merge_vocab","repo":"Suro-One/Hyena-Hierarchy","repo_kind":"listed","path":"hyena-split-memory.py","file_url":"https://github.com/Suro-One/Hyena-Hierarchy/blob/HEAD/hyena-split-memory.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3fbe2e426af0f8c"}},{"code_sha256_prefix":"5dd28d20c60f0c19","entry":"mul_sum","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"5dd28d20c60f0c19"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}