{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-practical-probabilistic-benchmark-for-ai","title":"A Practical Probabilistic Benchmark for AI Weather Models","arxiv_id":"2401.15305","date":"2024-01-27","proceeding":null,"authors":["Noah D. Brenowitz","Yair Cohen","Jaideep Pathak","Ankur Mahesh","Boris Bonev","Thorsten Kurth","Dale R. Durran","Peter Harrington","Michael S. Pritchard"],"abstract":"Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI weather models in a probabilistic sense, since variations in choice of ensemble initialization, definition of state, and noise injection methodology become confounding. Moreover, even obtaining ensemble forecast baselines is a substantial engineering challenge given the data volumes involved. We sidestep both problems by applying a decades-old idea -- lagged ensembles -- whereby an ensemble can be constructed from a moderately-sized library of deterministic forecasts. This allows the first parameter-free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. The results reveal that two leading AI weather models, i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though the former outperforms the latter in deterministic scoring. We also reveal how multiple time-step loss functions, which many data-driven weather models have employed, are counter-productive: they improve deterministic metrics at the cost of increased dissipation, deteriorating probabilistic skill. This is confirmed through ablations applied to a spherical Fourier Neural Operator (SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating effective resolution reveal it has a useful effect on ensemble dispersion relevant to achieving good ensemble calibration. We hope these and forthcoming insights from lagged ensembles can help guide the development of AI weather forecasts and have thus shared the diagnostic code.","url_abs":"https://arxiv.org/abs/2401.15305v2","url_pdf":"https://arxiv.org/pdf/2401.15305v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-practical-probabilistic-benchmark-for-ai","repo_url":"https://github.com/nvidia/earth2mip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"weather-forecasting","task_name":"Weather Forecasting"}],"methods":[{"method_slug":null,"method_name":"Library"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.15305","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.15305"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nvidia/earth2mip","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6250c8925253db10","entry":"brown_noise","repo":"nvidia/earth2mip","repo_kind":"official","path":"earth2mip/ensemble_utils.py","file_url":"https://github.com/nvidia/earth2mip/blob/HEAD/earth2mip/ensemble_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6250c8925253db10"}},{"code_sha256_prefix":"eff6c662c2b437e6","entry":"generate_noise_correlated","repo":"nvidia/earth2mip","repo_kind":"official","path":"earth2mip/ensemble_utils.py","file_url":"https://github.com/nvidia/earth2mip/blob/HEAD/earth2mip/ensemble_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eff6c662c2b437e6"}},{"code_sha256_prefix":"2cf8bccd7f94847a","entry":"get_times","repo":"nvidia/earth2mip","repo_kind":"official","path":"earth2mip/_cli_utils.py","file_url":"https://github.com/nvidia/earth2mip/blob/HEAD/earth2mip/_cli_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2cf8bccd7f94847a"}},{"code_sha256_prefix":"f710ae3fb9cf69b5","entry":"read_metrics","repo":"nvidia/earth2mip","repo_kind":"official","path":"earth2mip/forecast_metrics_io.py","file_url":"https://github.com/nvidia/earth2mip/blob/HEAD/earth2mip/forecast_metrics_io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f710ae3fb9cf69b5"}},{"code_sha256_prefix":"2f8a3da75df1323d","entry":"crps","repo":"nvidia/earth2mip","repo_kind":"official","path":"earth2mip/crps.py","file_url":"https://github.com/nvidia/earth2mip/blob/HEAD/earth2mip/crps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f8a3da75df1323d"}},{"code_sha256_prefix":"c9bd5eacf8c6848f","entry":"kcrps","repo":"nvidia/earth2mip","repo_kind":"official","path":"earth2mip/crps.py","file_url":"https://github.com/nvidia/earth2mip/blob/HEAD/earth2mip/crps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9bd5eacf8c6848f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}