{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sea-state-exchange-attention-for-high","title":"SEA: State-Exchange Attention for High-Fidelity Physics Based Transformers","arxiv_id":"2410.15495","date":"2024-10-20","proceeding":null,"authors":["Parsa Esmati","Amirhossein Dadashzadeh","Vahid Goodarzi","Nicolas Larrosa","Nicolò Grilli"],"abstract":"Current approaches using sequential networks have shown promise in estimating field variables for dynamical systems, but they are often limited by high rollout errors. The unresolved issue of rollout error accumulation results in unreliable estimations as the network predicts further into the future, with each step's error compounding and leading to an increase in inaccuracy. Here, we introduce the State-Exchange Attention (SEA) module, a novel transformer-based module enabling information exchange between encoded fields through multi-head cross-attention. The cross-field multidirectional information exchange design enables all state variables in the system to exchange information with one another, capturing physical relationships and symmetries between fields. Additionally, we introduce an efficient ViT-like mesh autoencoder to generate spatially coherent mesh embeddings for a large number of meshing cells. The SEA integrated transformer demonstrates the state-of-the-art rollout error compared to other competitive baselines. Specifically, we outperform PbGMR-GMUS Transformer-RealNVP and GMR-GMUS Transformer, with a reduction in error of 88% and 91%, respectively. Furthermore, we demonstrate that the SEA module alone can reduce errors by 97% for state variables that are highly dependent on other states of the system. The repository for this work is available at: https://github.com/ParsaEsmati/SEA","url_abs":"https://arxiv.org/abs/2410.15495v2","url_pdf":"https://arxiv.org/pdf/2410.15495v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sea-state-exchange-attention-for-high","repo_url":"https://github.com/parsaesmati/sea","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.15495","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.15495"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/parsaesmati/sea","reach":null}],"summary":{"ran":8,"ran_fixture":1,"ran_draft_wrong":1,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":13,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7d6282f4af9934f4","entry":"AdaLN","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d6282f4af9934f4"}},{"code_sha256_prefix":"944e9f9eb07f86a8","entry":"GaussianFourierProjection","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"944e9f9eb07f86a8"}},{"code_sha256_prefix":"69052bc6e30546ab","entry":"LayerNorm","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"69052bc6e30546ab"}},{"code_sha256_prefix":"feac008723761cd0","entry":"MLP","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"feac008723761cd0"}},{"code_sha256_prefix":"8a7b71e050983999","entry":"MaskedMultiHeadAttention","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a7b71e050983999"}},{"code_sha256_prefix":"ca93b4c2d02df885","entry":"MaskedMultiHeadCrossAttention","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca93b4c2d02df885"}},{"code_sha256_prefix":"cfe2061393c3ed76","entry":"MultiHeadCrossAttention","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cfe2061393c3ed76"}},{"code_sha256_prefix":"087f1d638aeab30c","entry":"PositionalEncoding","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"087f1d638aeab30c"}},{"code_sha256_prefix":"397ac9f96a533830","entry":"apply_rotary_emb","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"397ac9f96a533830"}},{"code_sha256_prefix":"67047903c6c8a8a5","entry":"load_and_convert","repo":"parsaesmati/sea","repo_kind":"official","path":"train/train_encoder.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/train/train_encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67047903c6c8a8a5"}},{"code_sha256_prefix":"e93cc5b705c2eb3b","entry":"precompute_freqs_cis","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e93cc5b705c2eb3b"}},{"code_sha256_prefix":"ad2ed8f5476af122","entry":"BaseBlockTemporal","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad2ed8f5476af122"}},{"code_sha256_prefix":"751944cbcb2f94ac","entry":"SEABlockTemporal","repo":"parsaesmati/sea","repo_kind":"official","path":"models/temporal.py","file_url":"https://github.com/parsaesmati/sea/blob/HEAD/models/temporal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"CC0-1.0","inline_ok":true,"mcp_get_code":{"code_sha256":"751944cbcb2f94ac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}