{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/earthformer-exploring-space-time-transformers","title":"Earthformer: Exploring Space-Time Transformers for Earth System Forecasting","arxiv_id":"2207.05833","date":"2022-07-12","proceeding":null,"authors":["Zhihan Gao","Xingjian Shi","Hao Wang","Yi Zhu","Yuyang Wang","Mu Li","Dit-yan Yeung"],"abstract":"Conventionally, Earth system (e.g., weather and climate) forecasting relies on numerical simulation with complex physical models and are hence both expensive in computation and demanding on domain expertise. With the explosive growth of the spatiotemporal Earth observation data in the past decade, data-driven models that apply Deep Learning (DL) are demonstrating impressive potential for various Earth system forecasting tasks. The Transformer as an emerging DL architecture, despite its broad success in other domains, has limited adoption in this area. In this paper, we propose Earthformer, a space-time Transformer for Earth system forecasting. Earthformer is based on a generic, flexible and efficient space-time attention block, named Cuboid Attention. The idea is to decompose the data into cuboids and apply cuboid-level self-attention in parallel. These cuboids are further connected with a collection of global vectors. We conduct experiments on the MovingMNIST dataset and a newly proposed chaotic N-body MNIST dataset to verify the effectiveness of cuboid attention and figure out the best design of Earthformer. Experiments on two real-world benchmarks about precipitation nowcasting and El Nino/Southern Oscillation (ENSO) forecasting show Earthformer achieves state-of-the-art performance. Code is available: https://github.com/amazon-science/earth-forecasting-transformer .","url_abs":"https://arxiv.org/abs/2207.05833v2","url_pdf":"https://arxiv.org/pdf/2207.05833v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"earthformer-exploring-space-time-transformers","repo_url":"https://github.com/amazon-science/earth-forecasting-transformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"earthformer-exploring-space-time-transformers","repo_url":"https://github.com/PaddlePaddle/PaddleScience","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"earth-observation","task_name":"Earth Observation"},{"task_slug":"earth-surface-forecasting","task_name":"Earth Surface Forecasting"},{"task_slug":"weather-forecasting","task_name":"Weather Forecasting"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/earth-surface-forecasting-on-earthnet2021-iid","task":"Earth Surface Forecasting","dataset":"EarthNet2021 IID Track","model":"Earthformer","rank_in_archive_order":1,"of":7,"metrics":{"EarthNetScore":"0.3425"},"uses_additional_data":false},{"leaderboard":"/sota/earth-surface-forecasting-on-earthnet2021-ood","task":"Earth Surface Forecasting","dataset":"EarthNet2021 OOD Track","model":"Earthformer","rank_in_archive_order":1,"of":7,"metrics":{"EarthNetScore":"0.3252"},"uses_additional_data":false},{"leaderboard":"/sota/weather-forecasting-on-sevir","task":"Weather Forecasting","dataset":"SEVIR","model":"Earthformer","rank_in_archive_order":2,"of":8,"metrics":{"MSE":"3.6957","mCSI":"0.4419"},"uses_additional_data":false},{"leaderboard":"/sota/weather-forecasting-on-sevir","task":"Weather Forecasting","dataset":"SEVIR","model":"ConvLSTM","rank_in_archive_order":3,"of":8,"metrics":{"MSE":"3.7532","mCSI":"0.4185"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.05833","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.05833"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PaddleScience","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-science/earth-forecasting-transformer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"767ad1f13b2a971b","entry":"compute_cuboid_self_attention_mask","repo":"amazon-science/earth-forecasting-transformer","repo_kind":"official","path":"src/earthformer/cuboid_transformer/cuboid_transformer.py","file_url":"https://github.com/amazon-science/earth-forecasting-transformer/blob/HEAD/src/earthformer/cuboid_transformer/cuboid_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"767ad1f13b2a971b"}},{"code_sha256_prefix":"fe7fc1f6b96b1356","entry":"compute_enso_score","repo":"amazon-science/earth-forecasting-transformer","repo_kind":"official","path":"src/earthformer/metrics/enso.py","file_url":"https://github.com/amazon-science/earth-forecasting-transformer/blob/HEAD/src/earthformer/metrics/enso.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe7fc1f6b96b1356"}},{"code_sha256_prefix":"bff27f9560afbd89","entry":"cuboid_reorder","repo":"amazon-science/earth-forecasting-transformer","repo_kind":"official","path":"src/earthformer/cuboid_transformer/cuboid_transformer.py","file_url":"https://github.com/amazon-science/earth-forecasting-transformer/blob/HEAD/src/earthformer/cuboid_transformer/cuboid_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bff27f9560afbd89"}},{"code_sha256_prefix":"84f8ab35b7d0471d","entry":"cuboid_reorder_reverse","repo":"amazon-science/earth-forecasting-transformer","repo_kind":"official","path":"src/earthformer/cuboid_transformer/cuboid_transformer.py","file_url":"https://github.com/amazon-science/earth-forecasting-transformer/blob/HEAD/src/earthformer/cuboid_transformer/cuboid_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"84f8ab35b7d0471d"}},{"code_sha256_prefix":"a3015a205293148b","entry":"full_attention","repo":"amazon-science/earth-forecasting-transformer","repo_kind":"official","path":"src/earthformer/cuboid_transformer/cuboid_transformer_patterns.py","file_url":"https://github.com/amazon-science/earth-forecasting-transformer/blob/HEAD/src/earthformer/cuboid_transformer/cuboid_transformer_patterns.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a3015a205293148b"}},{"code_sha256_prefix":"8c21cdd91cd386e3","entry":"get_activation","repo":"amazon-science/earth-forecasting-transformer","repo_kind":"official","path":"src/earthformer/cuboid_transformer/utils.py","file_url":"https://github.com/amazon-science/earth-forecasting-transformer/blob/HEAD/src/earthformer/cuboid_transformer/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c21cdd91cd386e3"}},{"code_sha256_prefix":"50a5a499347fcf81","entry":"get_norm_layer","repo":"amazon-science/earth-forecasting-transformer","repo_kind":"official","path":"src/earthformer/cuboid_transformer/utils.py","file_url":"https://github.com/amazon-science/earth-forecasting-transformer/blob/HEAD/src/earthformer/cuboid_transformer/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"50a5a499347fcf81"}},{"code_sha256_prefix":"50449b18203e5788","entry":"round_to","repo":"amazon-science/earth-forecasting-transformer","repo_kind":"official","path":"src/earthformer/cuboid_transformer/utils.py","file_url":"https://github.com/amazon-science/earth-forecasting-transformer/blob/HEAD/src/earthformer/cuboid_transformer/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"50449b18203e5788"}},{"code_sha256_prefix":"6bc7869a75f3e2db","entry":"self_axial","repo":"amazon-science/earth-forecasting-transformer","repo_kind":"official","path":"src/earthformer/cuboid_transformer/cuboid_transformer_patterns.py","file_url":"https://github.com/amazon-science/earth-forecasting-transformer/blob/HEAD/src/earthformer/cuboid_transformer/cuboid_transformer_patterns.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6bc7869a75f3e2db"}},{"code_sha256_prefix":"22d035b51d33b141","entry":"self_video_swin","repo":"amazon-science/earth-forecasting-transformer","repo_kind":"official","path":"src/earthformer/cuboid_transformer/cuboid_transformer_patterns.py","file_url":"https://github.com/amazon-science/earth-forecasting-transformer/blob/HEAD/src/earthformer/cuboid_transformer/cuboid_transformer_patterns.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"22d035b51d33b141"}},{"code_sha256_prefix":"dae75891e6023d74","entry":"sst_to_nino","repo":"amazon-science/earth-forecasting-transformer","repo_kind":"official","path":"src/earthformer/metrics/enso.py","file_url":"https://github.com/amazon-science/earth-forecasting-transformer/blob/HEAD/src/earthformer/metrics/enso.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dae75891e6023d74"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}