{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/chaos-meets-attention-transformers-for-large","title":"Chaos Meets Attention: Transformers for Large-Scale Dynamical Prediction","arxiv_id":"2504.20858","date":"2025-04-29","proceeding":null,"authors":["Yi He","Yiming Yang","Xiaoyuan Cheng","Hai Wang","Xiao Xue","Boli Chen","Yukun Hu"],"abstract":"Generating long-term trajectories of dissipative chaotic systems autoregressively is a highly challenging task. The inherent positive Lyapunov exponents amplify prediction errors over time. Many chaotic systems possess a crucial property - ergodicity on their attractors, which makes long-term prediction possible. State-of-the-art methods address ergodicity by preserving statistical properties using optimal transport techniques. However, these methods face scalability challenges due to the curse of dimensionality when matching distributions. To overcome this bottleneck, we propose a scalable transformer-based framework capable of stably generating long-term high-dimensional and high-resolution chaotic dynamics while preserving ergodicity. Our method is grounded in a physical perspective, revisiting the Von Neumann mean ergodic theorem to ensure the preservation of long-term statistics in the $\\mathcal{L}^2$ space. We introduce novel modifications to the attention mechanism, making the transformer architecture well-suited for learning large-scale chaotic systems. Compared to operator-based and transformer-based methods, our model achieves better performances across five metrics, from short-term prediction accuracy to long-term statistics. In addition to our methodological contributions, we introduce a new chaotic system benchmark: a machine learning dataset of 140$k$ snapshots of turbulent channel flow along with various evaluation metrics for both short- and long-term performances, which is well-suited for machine learning research on chaotic systems.","url_abs":"https://arxiv.org/abs/2504.20858v2","url_pdf":"https://arxiv.org/pdf/2504.20858v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"chaos-meets-attention-transformers-for-large","repo_url":"https://github.com/Hy23333/ChaosMeetsAttention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2504.20858","atlas_url":"https://app.syntology.ai/?focus=2504.20858","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.20858"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/neuraloperator/neuraloperator","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/gaurav71531/mwt-operator","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Hy23333/ChaosMeetsAttention","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":1,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"0a550f84606d37a9","entry":"compl_mul1d","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"0a550f84606d37a9"}},{"code_sha256_prefix":"afffefd34e1f0ff6","entry":"sparseKernelFT1d","repo":"gaurav71531/mwt-operator","repo_kind":"found_in_text","path":"models/models.py","file_url":"https://github.com/gaurav71531/mwt-operator/blob/HEAD/models/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"afffefd34e1f0ff6"}},{"code_sha256_prefix":"7a3a9223cdfdc6e7","entry":"MWT1d","repo":"gaurav71531/mwt-operator","repo_kind":"found_in_text","path":"models/models.py","file_url":"https://github.com/gaurav71531/mwt-operator/blob/HEAD/models/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a3a9223cdfdc6e7"}},{"code_sha256_prefix":"3575e83cfd1bac8f","entry":"MWT_CZ1d","repo":"gaurav71531/mwt-operator","repo_kind":"found_in_text","path":"models/models.py","file_url":"https://github.com/gaurav71531/mwt-operator/blob/HEAD/models/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3575e83cfd1bac8f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}