{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmst-vit-climate-change-aware-crop-yield","title":"MMST-ViT: Climate Change-aware Crop Yield Prediction via Multi-Modal Spatial-Temporal Vision Transformer","arxiv_id":"2309.09067","date":"2023-09-16","proceeding":"ICCV 2023 1","authors":["Fudong Lin","Summer Crawford","Kaleb Guillot","Yihe Zhang","Yan Chen","Xu Yuan","Li Chen","Shelby Williams","Robert Minvielle","Xiangming Xiao","Drew Gholson","Nicolas Ashwell","Tri Setiyono","Brenda Tubana","Lu Peng","Magdy Bayoumi","Nian-Feng Tzeng"],"abstract":"Precise crop yield prediction provides valuable information for agricultural planning and decision-making processes. However, timely predicting crop yields remains challenging as crop growth is sensitive to growing season weather variation and climate change. In this work, we develop a deep learning-based solution, namely Multi-Modal Spatial-Temporal Vision Transformer (MMST-ViT), for predicting crop yields at the county level across the United States, by considering the effects of short-term meteorological variations during the growing season and the long-term climate change on crops. Specifically, our MMST-ViT consists of a Multi-Modal Transformer, a Spatial Transformer, and a Temporal Transformer. The Multi-Modal Transformer leverages both visual remote sensing data and short-term meteorological data for modeling the effect of growing season weather variations on crop growth. The Spatial Transformer learns the high-resolution spatial dependency among counties for accurate agricultural tracking. The Temporal Transformer captures the long-range temporal dependency for learning the impact of long-term climate change on crops. Meanwhile, we also devise a novel multi-modal contrastive learning technique to pre-train our model without extensive human supervision. Hence, our MMST-ViT captures the impacts of both short-term weather variations and long-term climate change on crops by leveraging both satellite images and meteorological data. We have conducted extensive experiments on over 200 counties in the United States, with the experimental results exhibiting that our MMST-ViT outperforms its counterparts under three performance metrics of interest.","url_abs":"https://arxiv.org/abs/2309.09067v2","url_pdf":"https://arxiv.org/pdf/2309.09067v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmst-vit-climate-change-aware-crop-yield","repo_url":"https://github.com/fudong03/mmst-vit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"crop-yield-prediction","task_name":"Crop Yield Prediction"},{"task_slug":"decision-making","task_name":"Decision Making"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"spatial-transformer","method_name":"Spatial Transformer"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.09067","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.09067"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fudong03/mmst-vit","reach":null}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"231cddc23589ca5c","entry":"FeedForward","repo":"fudong03/mmst-vit","repo_kind":"official","path":"models_mmst_vit.py","file_url":"https://github.com/fudong03/mmst-vit/blob/HEAD/models_mmst_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"231cddc23589ca5c"}},{"code_sha256_prefix":"da3b4c554c894e11","entry":"SpatialAttention","repo":"fudong03/mmst-vit","repo_kind":"official","path":"models_mmst_vit.py","file_url":"https://github.com/fudong03/mmst-vit/blob/HEAD/models_mmst_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"da3b4c554c894e11"}},{"code_sha256_prefix":"705731cf1e54ec1f","entry":"SpatialTransformer","repo":"fudong03/mmst-vit","repo_kind":"official","path":"models_mmst_vit.py","file_url":"https://github.com/fudong03/mmst-vit/blob/HEAD/models_mmst_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"705731cf1e54ec1f"}},{"code_sha256_prefix":"eb09a0af031b731c","entry":"TemporalAttention","repo":"fudong03/mmst-vit","repo_kind":"official","path":"models_mmst_vit.py","file_url":"https://github.com/fudong03/mmst-vit/blob/HEAD/models_mmst_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"eb09a0af031b731c"}},{"code_sha256_prefix":"5b3304cb6896c505","entry":"TemporalTransformer","repo":"fudong03/mmst-vit","repo_kind":"official","path":"models_mmst_vit.py","file_url":"https://github.com/fudong03/mmst-vit/blob/HEAD/models_mmst_vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5b3304cb6896c505"}},{"code_sha256_prefix":"fc20cc4bbb81d7d9","entry":"MMST_ViT","repo":"fudong03/mmst-vit","repo_kind":"official","path":"models_mmst_vit.py","file_url":"https://github.com/fudong03/mmst-vit/blob/HEAD/models_mmst_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fc20cc4bbb81d7d9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}