{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/satmae-pre-training-transformers-for-temporal","title":"SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery","arxiv_id":"2207.08051","date":"2022-07-17","proceeding":null,"authors":["Yezhen Cong","Samar Khanna","Chenlin Meng","Patrick Liu","Erik Rozi","Yutong He","Marshall Burke","David B. Lobell","Stefano Ermon"],"abstract":"Unsupervised pre-training methods for large vision models have shown to enhance performance on downstream supervised tasks. Developing similar techniques for satellite imagery presents significant opportunities as unlabelled data is plentiful and the inherent temporal and multi-spectral structure provides avenues to further improve existing pre-training strategies. In this paper, we present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE). To leverage temporal information, we include a temporal embedding along with independently masking image patches across time. In addition, we demonstrate that encoding multi-spectral data as groups of bands with distinct spectral positional encodings is beneficial. Our approach yields strong improvements over previous state-of-the-art techniques, both in terms of supervised learning performance on benchmark datasets (up to $\\uparrow$ 7%), and transfer learning performance on downstream remote sensing tasks, including land cover classification (up to $\\uparrow$ 14%) and semantic segmentation. Code and data are available on the project website: https://sustainlab-group.github.io/SatMAE/","url_abs":"https://arxiv.org/abs/2207.08051v3","url_pdf":"https://arxiv.org/pdf/2207.08051v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"satmae-pre-training-transformers-for-temporal","repo_url":"https://github.com/sustainlab-group/SatMAE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"land-cover-classification","task_name":"Land Cover Classification"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"unsupervised-pre-training","task_name":"Unsupervised Pre-training"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.08051","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.08051"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sustainlab-group/SatMAE","reach":null}],"summary":{"unverified":2},"by_repo_kind":{"listed":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"40c2d10d3de426ed","entry":"MaskedAutoencoderViT","repo":"sustainlab-group/SatMAE","repo_kind":"listed","path":"models_mae_temporal.py","file_url":"https://github.com/sustainlab-group/SatMAE/blob/HEAD/models_mae_temporal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"40c2d10d3de426ed"}},{"code_sha256_prefix":"4321222177dba040","entry":"get_1d_sincos_pos_embed_from_grid_torch","repo":"sustainlab-group/SatMAE","repo_kind":"listed","path":"models_mae_temporal.py","file_url":"https://github.com/sustainlab-group/SatMAE/blob/HEAD/models_mae_temporal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4321222177dba040"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}