{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/xlstm-extended-long-short-term-memory","title":"xLSTM: Extended Long Short-Term Memory","arxiv_id":"2405.04517","date":"2024-05-07","proceeding":null,"authors":["Maximilian Beck","Korbinian Pöppel","Markus Spanring","Andreas Auer","Oleksandra Prudnikova","Michael Kopp","Günter Klambauer","Johannes Brandstetter","Sepp Hochreiter"],"abstract":"In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise a simple question: How far do we get in language modeling when scaling LSTMs to billions of parameters, leveraging the latest techniques from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we introduce exponential gating with appropriate normalization and stabilization techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that is fully parallelizable with a matrix memory and a covariance update rule. Integrating these LSTM extensions into residual block backbones yields xLSTM blocks that are then residually stacked into xLSTM architectures. Exponential gating and modified memory structures boost xLSTM capabilities to perform favorably when compared to state-of-the-art Transformers and State Space Models, both in performance and scaling.","url_abs":"https://arxiv.org/abs/2405.04517v2","url_pdf":"https://arxiv.org/pdf/2405.04517v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"xlstm-extended-long-short-term-memory","repo_url":"https://github.com/nx-ai/xlstm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"xlstm-extended-long-short-term-memory","repo_url":"https://github.com/gonzalopezgil/xlstm-ts","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"xlstm-extended-long-short-term-memory","repo_url":"https://github.com/nx-ai/mlstm_kernels","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"xlstm-extended-long-short-term-memory","repo_url":"https://github.com/nx-ai/xlstm-jax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"xlstm-extended-long-short-term-memory","repo_url":"https://github.com/smvorwerk/xlstm-cuda","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"mlstm","method_name":"mLSTM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.04517","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.04517"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nx-ai/xlstm","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nx-ai/xlstm-jax","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nx-ai/mlstm_kernels","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/smvorwerk/xlstm-cuda","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gonzalopezgil/xlstm-ts","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"listed":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c418ccbf439fb6c1","entry":"create_dataloader","repo":"gonzalopezgil/xlstm-ts","repo_kind":"listed","path":"src/ml/models/xlstm_ts/training.py","file_url":"https://github.com/gonzalopezgil/xlstm-ts/blob/HEAD/src/ml/models/xlstm_ts/training.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c418ccbf439fb6c1"}},{"code_sha256_prefix":"6ecfef88bb7757dc","entry":"display_metrics","repo":"gonzalopezgil/xlstm-ts","repo_kind":"listed","path":"src/ml/results/results.py","file_url":"https://github.com/gonzalopezgil/xlstm-ts/blob/HEAD/src/ml/results/results.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ecfef88bb7757dc"}},{"code_sha256_prefix":"eefa6a8362a8958c","entry":"evaluate_model","repo":"gonzalopezgil/xlstm-ts","repo_kind":"listed","path":"src/ml/models/xlstm_ts/training.py","file_url":"https://github.com/gonzalopezgil/xlstm-ts/blob/HEAD/src/ml/models/xlstm_ts/training.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eefa6a8362a8958c"}},{"code_sha256_prefix":"5a707ddedc88e249","entry":"get_logger","repo":"gonzalopezgil/xlstm-ts","repo_kind":"listed","path":"src/ml/utils/logger.py","file_url":"https://github.com/gonzalopezgil/xlstm-ts/blob/HEAD/src/ml/utils/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a707ddedc88e249"}},{"code_sha256_prefix":"41d97ba1265b3c73","entry":"inverse_normalise_data_xlstm","repo":"gonzalopezgil/xlstm-ts","repo_kind":"listed","path":"src/ml/models/xlstm_ts/preprocessing.py","file_url":"https://github.com/gonzalopezgil/xlstm-ts/blob/HEAD/src/ml/models/xlstm_ts/preprocessing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41d97ba1265b3c73"}},{"code_sha256_prefix":"24736fff7894ed00","entry":"normalise_data_xlstm","repo":"gonzalopezgil/xlstm-ts","repo_kind":"listed","path":"src/ml/models/xlstm_ts/preprocessing.py","file_url":"https://github.com/gonzalopezgil/xlstm-ts/blob/HEAD/src/ml/models/xlstm_ts/preprocessing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24736fff7894ed00"}},{"code_sha256_prefix":"a32843f8e0256e8a","entry":"show_results","repo":"gonzalopezgil/xlstm-ts","repo_kind":"listed","path":"src/ml/results/results.py","file_url":"https://github.com/gonzalopezgil/xlstm-ts/blob/HEAD/src/ml/results/results.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a32843f8e0256e8a"}},{"code_sha256_prefix":"5fec394cd964815b","entry":"train_model","repo":"gonzalopezgil/xlstm-ts","repo_kind":"listed","path":"src/ml/models/xlstm_ts/training.py","file_url":"https://github.com/gonzalopezgil/xlstm-ts/blob/HEAD/src/ml/models/xlstm_ts/training.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fec394cd964815b"}},{"code_sha256_prefix":"dc006911ff24188e","entry":"validate_date","repo":"gonzalopezgil/xlstm-ts","repo_kind":"listed","path":"src/gui/utils.py","file_url":"https://github.com/gonzalopezgil/xlstm-ts/blob/HEAD/src/gui/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc006911ff24188e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}