{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-02228","title":"xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity","arxiv_id":"2510.02228","date":"2025-10-02","proceeding":"ICLR","authors":["Maximilian Beck","Kajetan Schweighofer","Sebastian Böck","Sebastian Lehner","Sepp Hochreiter"],"abstract":"Scaling laws play a central role in the success of Large Language Models (LLMs), enabling the prediction of model performance relative to compute budgets prior to training. While Transformers have been the dominant architecture, recent alternatives such as xLSTM offer linear complexity with respect to context length while remaining competitive in the billion-parameter regime. We conduct a comparative investigation on the scaling behavior of Transformers and xLSTM along the following lines, providing insights to guide future model design and deployment. First, we study the scaling behavior for xLSTM in compute-optimal and over-training regimes using both IsoFLOP and parametric fit approaches on a wide range of model sizes (80M-7B) and number of training tokens (2B-2T). Second, we examine the dependence of optimal model sizes on context length, a pivotal aspect that was largely ignored in previous work. Finally, we analyze inference-time scaling characteristics. Our findings reveal that in typical LLM training and inference scenarios, xLSTM scales favorably compared to Transformers. Notably, xLSTM models consistently Pareto-dominate Transformer models, delivering lower cross-entropy loss for the same compute budget.","url_abs":"https://arxiv.org/abs/2510.02228","url_pdf":"https://arxiv.org/pdf/2510.02228","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.02228","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.02228"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/NX-AI/mlstm_kernels","reach":{"status":"ok","spdx":"NOASSERTION"}},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/NX-AI/xlstm-jax","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"unverified":19},"by_repo_kind":{"found_in_text":{"samples":19,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":19,"samples":[{"code_sha256_prefix":"2d94868c29f5e250","entry":"attention_causal_pt_cudnn","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/baselines/flash_attention/torch_sdp_attention.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/baselines/flash_attention/torch_sdp_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2d94868c29f5e250"}},{"code_sha256_prefix":"2bfbdc63b75b8758","entry":"attention_causal_pt_fa2","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/baselines/flash_attention/torch_sdp_attention.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/baselines/flash_attention/torch_sdp_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2bfbdc63b75b8758"}},{"code_sha256_prefix":"0c704f145e43d91a","entry":"attention_causal_pt_math","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/baselines/flash_attention/torch_sdp_attention.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/baselines/flash_attention/torch_sdp_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0c704f145e43d91a"}},{"code_sha256_prefix":"658eb63a92442898","entry":"class_to_name","repo":"NX-AI/xlstm-jax","repo_kind":"found_in_text","path":"xlstm_jax/import_utils.py","file_url":"https://github.com/NX-AI/xlstm-jax/blob/HEAD/xlstm_jax/import_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"658eb63a92442898"}},{"code_sha256_prefix":"b2ffb186a368be72","entry":"compute_total_model_flops","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/utils/flops/model_flops_computation.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/utils/flops/model_flops_computation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b2ffb186a368be72"}},{"code_sha256_prefix":"37e2df0514602d43","entry":"compute_total_model_flops_for_block_flops_dict","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/utils/flops/model_flops_computation.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/utils/flops/model_flops_computation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"37e2df0514602d43"}},{"code_sha256_prefix":"026ce928b14fa81f","entry":"contiguous","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/torch/utils.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/torch/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"026ce928b14fa81f"}},{"code_sha256_prefix":"3e55cc6a7427acae","entry":"dtype2str","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/jax/utils.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/jax/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3e55cc6a7427acae"}},{"code_sha256_prefix":"361348f3143f2432","entry":"dtype2str","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/torch/utils.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/torch/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"361348f3143f2432"}},{"code_sha256_prefix":"8ae078c843f8e319","entry":"get_stride","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/jax/stride_utils.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/jax/stride_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8ae078c843f8e319"}},{"code_sha256_prefix":"6560db241100ba72","entry":"get_strides","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/jax/stride_utils.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/jax/stride_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6560db241100ba72"}},{"code_sha256_prefix":"78c9d83eed016b5f","entry":"is_power_of_2","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/utils/kernels.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/utils/kernels.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"78c9d83eed016b5f"}},{"code_sha256_prefix":"1c08f5b6c2ddb6d0","entry":"resolve_import","repo":"NX-AI/xlstm-jax","repo_kind":"found_in_text","path":"xlstm_jax/import_utils.py","file_url":"https://github.com/NX-AI/xlstm-jax/blob/HEAD/xlstm_jax/import_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1c08f5b6c2ddb6d0"}},{"code_sha256_prefix":"e1a2912f3f776042","entry":"resolve_import_from_string","repo":"NX-AI/xlstm-jax","repo_kind":"found_in_text","path":"xlstm_jax/import_utils.py","file_url":"https://github.com/NX-AI/xlstm-jax/blob/HEAD/xlstm_jax/import_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e1a2912f3f776042"}},{"code_sha256_prefix":"bdab6beecbe21c54","entry":"to_numpy","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/jax/utils.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/jax/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"bdab6beecbe21c54"}},{"code_sha256_prefix":"613b28a130f51d5b","entry":"torch2triton_dtype","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/torch/utils.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/torch/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"613b28a130f51d5b"}},{"code_sha256_prefix":"96db97b5fd6e9fec","entry":"wrap_chunkwise__arbitrary_sequence_length","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/torch/kernel_wrappers.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/torch/kernel_wrappers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"96db97b5fd6e9fec"}},{"code_sha256_prefix":"9a7c16a025a91fbc","entry":"wrap_chunkwise__arbitrary_sequence_length_with_padding","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/torch/kernel_wrappers.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/torch/kernel_wrappers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9a7c16a025a91fbc"}},{"code_sha256_prefix":"c5dbe3df1807c498","entry":"wrap_chunkwise__pad_zeros","repo":"NX-AI/mlstm_kernels","repo_kind":"found_in_text","path":"mlstm_kernels/torch/kernel_wrappers.py","file_url":"https://github.com/NX-AI/mlstm_kernels/blob/HEAD/mlstm_kernels/torch/kernel_wrappers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c5dbe3df1807c498"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}