{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hyenadna-long-range-genomic-sequence-modeling","title":"HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution","arxiv_id":"2306.15794","date":"2023-06-27","proceeding":"NeurIPS 2023 11","authors":["Eric Nguyen","Michael Poli","Marjan Faizi","Armin Thomas","Callum Birch-Sykes","Michael Wornow","Aman Patel","Clayton Rabideau","Stefano Massaroli","Yoshua Bengio","Stefano Ermon","Stephen A. Baccus","Chris Ré"],"abstract":"Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data. On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points. Code at https://github.com/HazyResearch/hyena-dna.","url_abs":"https://arxiv.org/abs/2306.15794v2","url_pdf":"https://arxiv.org/pdf/2306.15794v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hyenadna-long-range-genomic-sequence-modeling","repo_url":"https://github.com/HazyResearch/hyena-dna","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"hyenadna-long-range-genomic-sequence-modeling","repo_url":"https://github.com/frederikkemarin/bend","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"hyenadna-long-range-genomic-sequence-modeling","repo_url":"https://github.com/jimmylihui/OpenGenome","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"hyenadna-long-range-genomic-sequence-modeling","repo_url":"https://github.com/jimmylihui/genbench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"4k","task_name":"4k"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.15794","atlas_url":"https://app.syntology.ai/?focus=2306.15794","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.15794"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/HazyResearch/hyena-dna","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jimmylihui/genbench","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/frederikkemarin/bend","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jimmylihui/OpenGenome","reach":null}],"summary":{"ran_draft_wrong":3,"ran":13,"ran_fixture":1,"unverified":11},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1},"listed":{"samples":19,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"3cd2f572b4362749","entry":"Activation","repo":"HazyResearch/hyena-dna","repo_kind":"official","path":"src/models/sequence/hyena.py","file_url":"https://github.com/HazyResearch/hyena-dna/blob/HEAD/src/models/sequence/hyena.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3cd2f572b4362749"}},{"code_sha256_prefix":"71c56d498ff19d96","entry":"ExponentialModulation","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"71c56d498ff19d96"}},{"code_sha256_prefix":"fb99304499c57d08","entry":"GPT2Embeddings","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"fb99304499c57d08"}},{"code_sha256_prefix":"b65ebc2556b1f0bd","entry":"HyenaFilter","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b65ebc2556b1f0bd"}},{"code_sha256_prefix":"916ff67d7cc5dec1","entry":"Laplace","repo":"HazyResearch/hyena-dna","repo_kind":"official","path":"src/models/sequence/hyena.py","file_url":"https://github.com/HazyResearch/hyena-dna/blob/HEAD/src/models/sequence/hyena.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"916ff67d7cc5dec1"}},{"code_sha256_prefix":"e9b64f401a36c5e0","entry":"LinearResidual","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e9b64f401a36c5e0"}},{"code_sha256_prefix":"2108a9e482ae37a6","entry":"MHA","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2108a9e482ae37a6"}},{"code_sha256_prefix":"d86ea40df8708d4d","entry":"Mlp","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d86ea40df8708d4d"}},{"code_sha256_prefix":"0e977f11203102fd","entry":"PositionalEmbedding","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0e977f11203102fd"}},{"code_sha256_prefix":"f745e0dbcc9bb059","entry":"SelfAttention","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f745e0dbcc9bb059"}},{"code_sha256_prefix":"d30504da4d509b40","entry":"SequenceDecoder","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d30504da4d509b40"}},{"code_sha256_prefix":"3f162814e9da788b","entry":"Sin","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3f162814e9da788b"}},{"code_sha256_prefix":"2c0fd690964e38f6","entry":"SquaredReLU","repo":"HazyResearch/hyena-dna","repo_kind":"official","path":"src/models/sequence/hyena.py","file_url":"https://github.com/HazyResearch/hyena-dna/blob/HEAD/src/models/sequence/hyena.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c0fd690964e38f6"}},{"code_sha256_prefix":"f930c490ea93d623","entry":"TransposedLN","repo":"HazyResearch/hyena-dna","repo_kind":"official","path":"src/models/sequence/hyena.py","file_url":"https://github.com/HazyResearch/hyena-dna/blob/HEAD/src/models/sequence/hyena.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f930c490ea93d623"}},{"code_sha256_prefix":"f365c364e5fb2475","entry":"create_mlp_cls","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f365c364e5fb2475"}},{"code_sha256_prefix":"452d521c52ebdf0f","entry":"laplace","repo":"HazyResearch/hyena-dna","repo_kind":"official","path":"src/models/sequence/hyena.py","file_url":"https://github.com/HazyResearch/hyena-dna/blob/HEAD/src/models/sequence/hyena.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"452d521c52ebdf0f"}},{"code_sha256_prefix":"5dd28d20c60f0c19","entry":"mul_sum","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"5dd28d20c60f0c19"}},{"code_sha256_prefix":"15e87d04566e903e","entry":"Block","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"15e87d04566e903e"}},{"code_sha256_prefix":"1a0693ded9212e71","entry":"HyenaDNAModel","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1a0693ded9212e71"}},{"code_sha256_prefix":"bdfcfc1b78dd3774","entry":"HyenaOperator","repo":"HazyResearch/hyena-dna","repo_kind":"official","path":"src/models/sequence/hyena.py","file_url":"https://github.com/HazyResearch/hyena-dna/blob/HEAD/src/models/sequence/hyena.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdfcfc1b78dd3774"}},{"code_sha256_prefix":"7b17a3ba43c38567","entry":"HyenaOperator","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"7b17a3ba43c38567"}},{"code_sha256_prefix":"989ba1bad54bbbc8","entry":"LMBackbone","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"989ba1bad54bbbc8"}},{"code_sha256_prefix":"8ae5fabb54f1ee70","entry":"_init_weights","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8ae5fabb54f1ee70"}},{"code_sha256_prefix":"b37e90eb829295dc","entry":"auto_assign_attrs","repo":"HazyResearch/hyena-dna","repo_kind":"official","path":"src/models/sequence/hyena.py","file_url":"https://github.com/HazyResearch/hyena-dna/blob/HEAD/src/models/sequence/hyena.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b37e90eb829295dc"}},{"code_sha256_prefix":"78db10ce14ffa95c","entry":"create_block","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"78db10ce14ffa95c"}},{"code_sha256_prefix":"fe93a5054c8dd38f","entry":"create_mixer_cls","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"fe93a5054c8dd38f"}},{"code_sha256_prefix":"41b32f57fc8ec1e5","entry":"fftconv","repo":"frederikkemarin/bend","repo_kind":"listed","path":"bend/models/hyena_dna.py","file_url":"https://github.com/frederikkemarin/bend/blob/HEAD/bend/models/hyena_dna.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"41b32f57fc8ec1e5"}},{"code_sha256_prefix":"ef3e1ed020307a6f","entry":"instantiate","repo":"HazyResearch/hyena-dna","repo_kind":"official","path":"src/models/sequence/hyena.py","file_url":"https://github.com/HazyResearch/hyena-dna/blob/HEAD/src/models/sequence/hyena.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef3e1ed020307a6f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}