{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-efficient-masked-autoencoder-learning","title":"Scaling Efficient Masked Image Modeling on Large Remote Sensing Dataset","arxiv_id":"2406.11933","date":"2024-06-17","proceeding":null,"authors":["Fengxiang Wang","Hongzhen Wang","Di Wang","Zonghao Guo","Zhenyu Zhong","Long Lan","Jing Zhang","Zhiyuan Liu","Maosong Sun"],"abstract":"Masked Image Modeling (MIM) has become an essential method for building foundational visual models in remote sensing (RS). However, the limitations in size and diversity of existing RS datasets restrict the ability of MIM methods to learn generalizable representations. Additionally, conventional MIM techniques, which require reconstructing all tokens, introduce unnecessary computational overhead. To address these issues, we present a new pre-training pipeline for RS models, featuring the creation of a large-scale RS dataset and an efficient MIM approach. We curated a high-quality dataset named OpticalRS-13M by collecting publicly available RS datasets and processing them through exclusion, slicing, and deduplication. OpticalRS-13M comprises 13 million optical images covering various RS tasks, such as object detection and pixel segmentation. To enhance efficiency, we propose SelectiveMAE, a pre-training method that dynamically encodes and reconstructs semantically rich patch tokens, thereby reducing the inefficiencies of traditional MIM models caused by redundant background pixels in RS images. Extensive experiments demonstrate that OpticalRS-13M significantly improves classification, detection, and segmentation performance, while SelectiveMAE increases training efficiency over 2 times. This highlights the effectiveness and scalability of our pipeline in developing RS foundational models.","url_abs":"https://arxiv.org/abs/2406.11933v4","url_pdf":"https://arxiv.org/pdf/2406.11933v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-efficient-masked-autoencoder-learning","repo_url":"https://github.com/Fengxiang23/SelectiveMAE","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"aerial-scene-classification","task_name":"Aerial Scene Classification"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-in-aerial-images","task_name":"Object Detection In Aerial Images"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":null,"task_name":"Sence Classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"mae","method_name":"MAE"},{"method_slug":"mim","method_name":"MIM"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/object-detection-in-aerial-images-on-dior","task":"Object Detection In Aerial Images","dataset":"DIOR","model":"SelectiveMAE+ViT-B","rank_in_archive_order":4,"of":4,"metrics":{"AP50":"77.80"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-loveda","task":"Semantic Segmentation","dataset":"LoveDA","model":"SelectiveMAE+ViT-L","rank_in_archive_order":5,"of":19,"metrics":{"Category mIoU":"54.31"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-spacenet-1","task":"Semantic Segmentation","dataset":"SpaceNet 1","model":"SelectiveMAE+ViT-B","rank_in_archive_order":4,"of":10,"metrics":{"Mean IoU":"79.50"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.11933","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.11933"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Fengxiang23/SelectiveMAE","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"519bbbff76c0ed7c","entry":"build_dataset","repo":"Fengxiang23/SelectiveMAE","repo_kind":"official","path":"SelectiveMAE/util/datasets.py","file_url":"https://github.com/Fengxiang23/SelectiveMAE/blob/HEAD/SelectiveMAE/util/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"519bbbff76c0ed7c"}},{"code_sha256_prefix":"ae15999500b3952c","entry":"build_transform","repo":"Fengxiang23/SelectiveMAE","repo_kind":"official","path":"SelectiveMAE/util/datasets.py","file_url":"https://github.com/Fengxiang23/SelectiveMAE/blob/HEAD/SelectiveMAE/util/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae15999500b3952c"}},{"code_sha256_prefix":"89edf5372214b57d","entry":"get_layer_id_for_vit","repo":"Fengxiang23/SelectiveMAE","repo_kind":"official","path":"SelectiveMAE/util/lr_decay.py","file_url":"https://github.com/Fengxiang23/SelectiveMAE/blob/HEAD/SelectiveMAE/util/lr_decay.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89edf5372214b57d"}},{"code_sha256_prefix":"8baf25cef41cba16","entry":"param_groups_lrd","repo":"Fengxiang23/SelectiveMAE","repo_kind":"official","path":"SelectiveMAE/util/lr_decay.py","file_url":"https://github.com/Fengxiang23/SelectiveMAE/blob/HEAD/SelectiveMAE/util/lr_decay.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8baf25cef41cba16"}},{"code_sha256_prefix":"09f4e10c77016d7e","entry":"concat_all_gather","repo":"Fengxiang23/SelectiveMAE","repo_kind":"official","path":"SelectiveMAE/infobatch.py","file_url":"https://github.com/Fengxiang23/SelectiveMAE/blob/HEAD/SelectiveMAE/infobatch.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09f4e10c77016d7e"}},{"code_sha256_prefix":"1a3e01d4552a3b11","entry":"handle_flash_attn","repo":"Fengxiang23/SelectiveMAE","repo_kind":"official","path":"SelectiveMAE/transformer_utils.py","file_url":"https://github.com/Fengxiang23/SelectiveMAE/blob/HEAD/SelectiveMAE/transformer_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a3e01d4552a3b11"}},{"code_sha256_prefix":"4b0f67bb535e2dd1","entry":"info_hack_indices","repo":"Fengxiang23/SelectiveMAE","repo_kind":"official","path":"SelectiveMAE/infobatch.py","file_url":"https://github.com/Fengxiang23/SelectiveMAE/blob/HEAD/SelectiveMAE/infobatch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b0f67bb535e2dd1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}