{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bio-xlstm-generative-modeling-representation","title":"Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences","arxiv_id":"2411.04165","date":"2024-11-06","proceeding":null,"authors":["Niklas Schmidinger","Lisa Schneckenreiter","Philipp Seidl","Johannes Schimunek","Pieter-Jan Hoedt","Johannes Brandstetter","Andreas Mayr","Sohvi Luukkonen","Sepp Hochreiter","Günter Klambauer"],"abstract":"Language models for biological and chemical sequences enable crucial applications such as drug discovery, protein engineering, and precision medicine. Currently, these language models are predominantly based on Transformer architectures. While Transformers have yielded impressive results, their quadratic runtime dependency on the sequence length complicates their use for long genomic sequences and in-context learning on proteins and chemical sequences. Recently, the recurrent xLSTM architecture has been shown to perform favorably compared to Transformers and modern state-space model (SSM) architectures in the natural language domain. Similar to SSMs, xLSTMs have a linear runtime dependency on the sequence length and allow for constant-memory decoding at inference time, which makes them prime candidates for modeling long-range dependencies in biological and chemical sequences. In this work, we tailor xLSTM towards these domains and propose a suite of architectural variants called Bio-xLSTM. Extensive experiments in three large domains, genomics, proteins, and chemistry, were performed to assess xLSTM's ability to model biological and chemical sequences. The results show that models based on Bio-xLSTM a) can serve as proficient generative models for DNA, protein, and chemical sequences, b) learn rich representations for those modalities, and c) can perform in-context learning for proteins and small molecules.","url_abs":"https://arxiv.org/abs/2411.04165v1","url_pdf":"https://arxiv.org/pdf/2411.04165v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bio-xlstm-generative-modeling-representation","repo_url":"https://github.com/ml-jku/chem-xlstm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"bio-xlstm-generative-modeling-representation","repo_url":"https://github.com/ml-jku/dna-xlstm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"bio-xlstm-generative-modeling-representation","repo_url":"https://github.com/ml-jku/prot-xlstm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"drug-discovery","task_name":"Drug Discovery"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.04165","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.04165"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-jku/dna-xlstm","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-jku/chem-xlstm","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ml-jku/prot-xlstm","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"ran_violates":1,"unverified":26},"by_repo_kind":{"official":{"samples":29,"ran":3,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"705d17e5f5c0489b","entry":"bias_linspace_init_","repo":"ml-jku/dna-xlstm","repo_kind":"official","path":"models/dna_xlstm/xlstm/components/init.py","file_url":"https://github.com/ml-jku/dna-xlstm/blob/HEAD/models/dna_xlstm/xlstm/components/init.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"705d17e5f5c0489b"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"ml-jku/dna-xlstm","repo_kind":"official","path":"models/dna_xlstm/xlstm/xlstm_block_stack.py","file_url":"https://github.com/ml-jku/dna-xlstm/blob/HEAD/models/dna_xlstm/xlstm/xlstm_block_stack.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"1f3623588b61ca31","entry":"krylov","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/module_library/krylov.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/module_library/krylov.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f3623588b61ca31"}},{"code_sha256_prefix":"ac829870c326eb73","entry":"apply_rotary_emb","repo":"ml-jku/prot-xlstm","repo_kind":"official","path":"protxlstm/xlstm/components/rotary_position.py","file_url":"https://github.com/ml-jku/prot-xlstm/blob/HEAD/protxlstm/xlstm/components/rotary_position.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac829870c326eb73"}},{"code_sha256_prefix":"bf0be6c26af1d44e","entry":"cauchy_naive","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/module_library/cauchy.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/module_library/cauchy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bf0be6c26af1d44e"}},{"code_sha256_prefix":"5688929ec94f213a","entry":"combination","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/module_library/dplr.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/module_library/dplr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5688929ec94f213a"}},{"code_sha256_prefix":"2a573b08e6c410c8","entry":"compute_freqs_cis","repo":"ml-jku/prot-xlstm","repo_kind":"official","path":"protxlstm/xlstm/components/rotary_position.py","file_url":"https://github.com/ml-jku/prot-xlstm/blob/HEAD/protxlstm/xlstm/components/rotary_position.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a573b08e6c410c8"}},{"code_sha256_prefix":"683dd071b7bcb7c7","entry":"conditional_decorator","repo":"ml-jku/dna-xlstm","repo_kind":"official","path":"models/dna_xlstm/xlstm/components/util.py","file_url":"https://github.com/ml-jku/dna-xlstm/blob/HEAD/models/dna_xlstm/xlstm/components/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"683dd071b7bcb7c7"}},{"code_sha256_prefix":"25766cf2cb87920f","entry":"conv1d_step","repo":"ml-jku/prot-xlstm","repo_kind":"official","path":"protxlstm/xlstm/components/conv.py","file_url":"https://github.com/ml-jku/prot-xlstm/blob/HEAD/protxlstm/xlstm/components/conv.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25766cf2cb87920f"}},{"code_sha256_prefix":"b52fb76a05e95adb","entry":"create_dataloader","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/dataloaders.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/dataloaders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b52fb76a05e95adb"}},{"code_sha256_prefix":"7c3fe064733e56b3","entry":"dplr","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/module_library/dplr.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/module_library/dplr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7c3fe064733e56b3"}},{"code_sha256_prefix":"e6845c8558af2604","entry":"embed_c2r","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/module_library/hippo.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/module_library/hippo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e6845c8558af2604"}},{"code_sha256_prefix":"ed05db9cecbe4f40","entry":"get_act_fn","repo":"ml-jku/dna-xlstm","repo_kind":"official","path":"models/dna_xlstm/xlstm/components/feedforward.py","file_url":"https://github.com/ml-jku/dna-xlstm/blob/HEAD/models/dna_xlstm/xlstm/components/feedforward.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed05db9cecbe4f40"}},{"code_sha256_prefix":"3b07afcb40c29e2a","entry":"get_mlstm_inner_dim","repo":"ml-jku/dna-xlstm","repo_kind":"official","path":"models/dna_xlstm/xlstm/xlstm_block_stack.py","file_url":"https://github.com/ml-jku/dna-xlstm/blob/HEAD/models/dna_xlstm/xlstm/xlstm_block_stack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3b07afcb40c29e2a"}},{"code_sha256_prefix":"b060864a3e29a600","entry":"krylov_sequential","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/module_library/krylov.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/module_library/krylov.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b060864a3e29a600"}},{"code_sha256_prefix":"1fb8af0f651b45c7","entry":"learn_label_encoding","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/fasta_utils.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/fasta_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1fb8af0f651b45c7"}},{"code_sha256_prefix":"4208c567af7da5a8","entry":"load_file","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/evaluate_cond_gen.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/evaluate_cond_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4208c567af7da5a8"}},{"code_sha256_prefix":"df4c6239424d356d","entry":"load_model","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/evaluate_cond_gen.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/evaluate_cond_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df4c6239424d356d"}},{"code_sha256_prefix":"8a748a919db6ec4d","entry":"load_smiles_from_zip","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/metrics.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a748a919db6ec4d"}},{"code_sha256_prefix":"5df4043c60e49613","entry":"power","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/module_library/krylov.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/module_library/krylov.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5df4043c60e49613"}},{"code_sha256_prefix":"2bf924786015498d","entry":"precompute_freqs_cis","repo":"ml-jku/dna-xlstm","repo_kind":"official","path":"models/dna_xlstm/xlstm/xlstm_block_stack.py","file_url":"https://github.com/ml-jku/dna-xlstm/blob/HEAD/models/dna_xlstm/xlstm/xlstm_block_stack.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2bf924786015498d"}},{"code_sha256_prefix":"433b3d872e02032c","entry":"rank_correction","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/module_library/hippo.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/module_library/hippo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"433b3d872e02032c"}},{"code_sha256_prefix":"6e10ab4b4709a633","entry":"read_fastas","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/fasta_utils.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/fasta_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e10ab4b4709a633"}},{"code_sha256_prefix":"3a70ec6e0ac819b2","entry":"round_to_multiple","repo":"ml-jku/dna-xlstm","repo_kind":"official","path":"models/dna_xlstm/xlstm/components/util.py","file_url":"https://github.com/ml-jku/dna-xlstm/blob/HEAD/models/dna_xlstm/xlstm/components/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3a70ec6e0ac819b2"}},{"code_sha256_prefix":"c0dc66eb84a1479a","entry":"segment_fasta","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/fasta_utils.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/fasta_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c0dc66eb84a1479a"}},{"code_sha256_prefix":"cfe16cc8552a2332","entry":"small_init_init_","repo":"ml-jku/dna-xlstm","repo_kind":"official","path":"models/dna_xlstm/xlstm/components/init.py","file_url":"https://github.com/ml-jku/dna-xlstm/blob/HEAD/models/dna_xlstm/xlstm/components/init.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cfe16cc8552a2332"}},{"code_sha256_prefix":"a59eccfb252a06f0","entry":"ssm","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/module_library/dplr.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/module_library/dplr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a59eccfb252a06f0"}},{"code_sha256_prefix":"34b2aa570368722c","entry":"transition","repo":"ml-jku/chem-xlstm","repo_kind":"official","path":"chemxlstm/module_library/hippo.py","file_url":"https://github.com/ml-jku/chem-xlstm/blob/HEAD/chemxlstm/module_library/hippo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34b2aa570368722c"}},{"code_sha256_prefix":"431c81ae48403967","entry":"wang_init_","repo":"ml-jku/dna-xlstm","repo_kind":"official","path":"models/dna_xlstm/xlstm/components/init.py","file_url":"https://github.com/ml-jku/dna-xlstm/blob/HEAD/models/dna_xlstm/xlstm/components/init.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"431c81ae48403967"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}