{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-transformers-pre-training-for","title":"Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery","arxiv_id":"2403.05419","date":"2024-03-08","proceeding":"CVPR 2024 1","authors":["Mubashir Noman","Muzammal Naseer","Hisham Cholakkal","Rao Muhammad Anwar","Salman Khan","Fahad Shahbaz Khan"],"abstract":"Recent advances in unsupervised learning have demonstrated the ability of large vision models to achieve promising results on downstream tasks by pre-training on large amount of unlabelled data. Such pre-training techniques have also been explored recently in the remote sensing domain due to the availability of large amount of unlabelled data. Different from standard natural image datasets, remote sensing data is acquired from various sensor technologies and exhibit diverse range of scale variations as well as modalities. Existing satellite image pre-training methods either ignore the scale information present in the remote sensing imagery or restrict themselves to use only a single type of data modality. In this paper, we re-visit transformers pre-training and leverage multi-scale information that is effectively utilized with multiple modalities. Our proposed approach, named SatMAE++, performs multi-scale pre-training and utilizes convolution based upsampling blocks to reconstruct the image at higher scales making it extensible to include more scales. Compared to existing works, the proposed SatMAE++ with multi-scale pre-training is equally effective for both optical as well as multi-spectral imagery. Extensive experiments on six datasets reveal the merits of proposed contributions, leading to state-of-the-art performance on all datasets. SatMAE++ achieves mean average precision (mAP) gain of 2.5\\% for multi-label classification task on BigEarthNet dataset. Our code and pre-trained models are available at \\url{https://github.com/techmn/satmae_pp}.","url_abs":"https://arxiv.org/abs/2403.05419v1","url_pdf":"https://arxiv.org/pdf/2403.05419v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-transformers-pre-training-for","repo_url":"https://github.com/techmn/satmae_pp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"multi-label-classification-2","task_name":"MUlTI-LABEL-ClASSIFICATION"},{"task_slug":"multi-label-classification","task_name":"Multi-Label Classification"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.05419","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.05419"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/techmn/satmae_pp","reach":null}],"summary":{"ran":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f7cde5f401e45872","entry":"LayerNorm","repo":"techmn/satmae_pp","repo_kind":"official","path":"models_mae_group_channels.py","file_url":"https://github.com/techmn/satmae_pp/blob/HEAD/models_mae_group_channels.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f7cde5f401e45872"}},{"code_sha256_prefix":"2af6dbba2ec2b3de","entry":"MaskedAutoencoderGroupChannelViT","repo":"techmn/satmae_pp","repo_kind":"official","path":"models_mae_group_channels.py","file_url":"https://github.com/techmn/satmae_pp/blob/HEAD/models_mae_group_channels.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2af6dbba2ec2b3de"}},{"code_sha256_prefix":"df04f82fc84f53c2","entry":"ResidualBlock","repo":"techmn/satmae_pp","repo_kind":"official","path":"models_mae_group_channels.py","file_url":"https://github.com/techmn/satmae_pp/blob/HEAD/models_mae_group_channels.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df04f82fc84f53c2"}},{"code_sha256_prefix":"79d85f1a4a3e06f9","entry":"UpsampleBlock","repo":"techmn/satmae_pp","repo_kind":"official","path":"models_mae_group_channels.py","file_url":"https://github.com/techmn/satmae_pp/blob/HEAD/models_mae_group_channels.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"79d85f1a4a3e06f9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}