{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/analyzing-and-exploring-training-recipes-for","title":"Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction","arxiv_id":"2404.19630","date":"2024-04-30","proceeding":null,"authors":["Jared D. Willard","Peter Harrington","Shashank Subramanian","Ankur Mahesh","Travis A. O'Brien","William D. Collins"],"abstract":"The rapid rise of deep learning (DL) in numerical weather prediction (NWP) has led to a proliferation of models which forecast atmospheric variables with comparable or superior skill than traditional physics-based NWP. However, among these leading DL models, there is a wide variance in both the training settings and architecture used. Further, the lack of thorough ablation studies makes it hard to discern which components are most critical to success. In this work, we show that it is possible to attain high forecast skill even with relatively off-the-shelf architectures, simple training procedures, and moderate compute budgets. Specifically, we train a minimally modified SwinV2 transformer on ERA5 data, and find that it attains superior forecast skill when compared against IFS. We present some ablations on key aspects of the training pipeline, exploring different loss functions, model sizes and depths, and multi-step fine-tuning to investigate their effect. We also examine the model performance with metrics beyond the typical ACC and RMSE, and investigate how the performance scales with model size.","url_abs":"https://arxiv.org/abs/2404.19630v1","url_pdf":"https://arxiv.org/pdf/2404.19630v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"analyzing-and-exploring-training-recipes-for","repo_url":"https://github.com/nersc/swin_v2_weather","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.19630","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.19630"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nersc/swin_v2_weather","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f575ba978436bea7","entry":"bchw_to_bhwc","repo":"nersc/swin_v2_weather","repo_kind":"official","path":"networks/swinv2_global.py","file_url":"https://github.com/nersc/swin_v2_weather/blob/HEAD/networks/swinv2_global.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f575ba978436bea7"}},{"code_sha256_prefix":"834245849579cf2f","entry":"bhwc_to_bchw","repo":"nersc/swin_v2_weather","repo_kind":"official","path":"networks/swinv2_global.py","file_url":"https://github.com/nersc/swin_v2_weather/blob/HEAD/networks/swinv2_global.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"834245849579cf2f"}},{"code_sha256_prefix":"a7534cc9bd54f268","entry":"get_land_mask","repo":"nersc/swin_v2_weather","repo_kind":"official","path":"utils/conditioning_inputs.py","file_url":"https://github.com/nersc/swin_v2_weather/blob/HEAD/utils/conditioning_inputs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7534cc9bd54f268"}},{"code_sha256_prefix":"de142677a0f2996a","entry":"interpolate","repo":"nersc/swin_v2_weather","repo_kind":"official","path":"utils/img_utils.py","file_url":"https://github.com/nersc/swin_v2_weather/blob/HEAD/utils/img_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"de142677a0f2996a"}},{"code_sha256_prefix":"ab094c14f4c11716","entry":"is_leap_year","repo":"nersc/swin_v2_weather","repo_kind":"official","path":"utils/data_loader_era5.py","file_url":"https://github.com/nersc/swin_v2_weather/blob/HEAD/utils/data_loader_era5.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab094c14f4c11716"}},{"code_sha256_prefix":"4fc3dc334552287c","entry":"vis","repo":"nersc/swin_v2_weather","repo_kind":"official","path":"utils/img_utils.py","file_url":"https://github.com/nersc/swin_v2_weather/blob/HEAD/utils/img_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4fc3dc334552287c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}