{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/chaosbench-a-multi-channel-physics-based","title":"ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction","arxiv_id":"2402.00712","date":"2024-02-01","proceeding":null,"authors":["Juan Nathaniel","Yongquan Qu","Tung Nguyen","Sungduk Yu","Julius Busecke","Aditya Grover","Pierre Gentine"],"abstract":"Accurate prediction of climate in the subseasonal-to-seasonal scale is crucial for disaster preparedness and robust decision making amidst climate change. Yet, forecasting beyond the weather timescale is challenging because it deals with problems other than initial condition, including boundary interaction, butterfly effect, and our inherent lack of physical understanding. At present, existing benchmarks tend to have shorter forecasting range of up-to 15 days, do not include a wide range of operational baselines, and lack physics-based constraints for explainability. Thus, we propose ChaosBench, a challenging benchmark to extend the predictability range of data-driven weather emulators to S2S timescale. First, ChaosBench is comprised of variables beyond the typical surface-atmospheric ERA5 to also include ocean, ice, and land reanalysis products that span over 45 years to allow for full Earth system emulation that respects boundary conditions. We also propose physics-based, in addition to deterministic and probabilistic metrics, to ensure a physically-consistent ensemble that accounts for butterfly effect. Furthermore, we evaluate on a diverse set of physics-based forecasts from four national weather agencies as baselines to our data-driven counterpart such as ViT/ClimaX, PanguWeather, GraphCast, and FourCastNetV2. Overall, we find methods originally developed for weather-scale applications fail on S2S task: their performance simply collapse to an unskilled climatology. Nonetheless, we outline and demonstrate several strategies that can extend the predictability range of existing weather emulators, including the use of ensembles, robust control of error propagation, and the use of physics-informed models. Our benchmark, datasets, and instructions are available at https://leap-stc.github.io/ChaosBench.","url_abs":"https://arxiv.org/abs/2402.00712v5","url_pdf":"https://arxiv.org/pdf/2402.00712v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"chaosbench-a-multi-channel-physics-based","repo_url":"https://github.com/leap-stc/ChaosBench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[{"slug":"chaosbench","name":"ChaosBench","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.00712","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.00712"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/leap-stc/ChaosBench","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/ecmwf-lab/ai-models","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_violates":2,"ran":9},"by_repo_kind":{"official":{"samples":9,"ran":9,"repositories":1},"found_in_text":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"1f7c550e65142ef8","entry":"cast_tuple","repo":"leap-stc/ChaosBench","repo_kind":"official","path":"chaosbench/models/vit.py","file_url":"https://github.com/leap-stc/ChaosBench/blob/HEAD/chaosbench/models/vit.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1f7c550e65142ef8"}},{"code_sha256_prefix":"92f1a3fa77bf2251","entry":"conv2d_bn_relu","repo":"leap-stc/ChaosBench","repo_kind":"official","path":"chaosbench/models/ae.py","file_url":"https://github.com/leap-stc/ChaosBench/blob/HEAD/chaosbench/models/ae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"92f1a3fa77bf2251"}},{"code_sha256_prefix":"143f63c99701b6a9","entry":"conv_block","repo":"leap-stc/ChaosBench","repo_kind":"official","path":"chaosbench/models/cnn.py","file_url":"https://github.com/leap-stc/ChaosBench/blob/HEAD/chaosbench/models/cnn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"143f63c99701b6a9"}},{"code_sha256_prefix":"3cd9a84568633114","entry":"convert_time","repo":"leap-stc/ChaosBench","repo_kind":"official","path":"chaosbench/utils.py","file_url":"https://github.com/leap-stc/ChaosBench/blob/HEAD/chaosbench/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3cd9a84568633114"}},{"code_sha256_prefix":"451d321a21300bea","entry":"deconv_relu","repo":"leap-stc/ChaosBench","repo_kind":"official","path":"chaosbench/models/ae.py","file_url":"https://github.com/leap-stc/ChaosBench/blob/HEAD/chaosbench/models/ae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"451d321a21300bea"}},{"code_sha256_prefix":"1d3a3f04669fabe7","entry":"deconv_sigmoid","repo":"leap-stc/ChaosBench","repo_kind":"official","path":"chaosbench/models/ae.py","file_url":"https://github.com/leap-stc/ChaosBench/blob/HEAD/chaosbench/models/ae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"1d3a3f04669fabe7"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"leap-stc/ChaosBench","repo_kind":"official","path":"chaosbench/models/vit.py","file_url":"https://github.com/leap-stc/ChaosBench/blob/HEAD/chaosbench/models/vit.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"b12e0af56d8cccb1","entry":"interpolate","repo":"leap-stc/ChaosBench","repo_kind":"official","path":"eval_direct.py","file_url":"https://github.com/leap-stc/ChaosBench/blob/HEAD/eval_direct.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b12e0af56d8cccb1"}},{"code_sha256_prefix":"9dd689a970ea2365","entry":"peek","repo":"ecmwf-lab/ai-models","repo_kind":"found_in_text","path":"src/ai_models/checkpoint.py","file_url":"https://github.com/ecmwf-lab/ai-models/blob/HEAD/src/ai_models/checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9dd689a970ea2365"}},{"code_sha256_prefix":"93ab76178df79a49","entry":"tidy","repo":"ecmwf-lab/ai-models","repo_kind":"found_in_text","path":"src/ai_models/checkpoint.py","file_url":"https://github.com/ecmwf-lab/ai-models/blob/HEAD/src/ai_models/checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"93ab76178df79a49"}},{"code_sha256_prefix":"362233a242a189a5","entry":"upconv_block","repo":"leap-stc/ChaosBench","repo_kind":"official","path":"chaosbench/models/cnn.py","file_url":"https://github.com/leap-stc/ChaosBench/blob/HEAD/chaosbench/models/cnn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"362233a242a189a5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}