{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-empirical-study-of-remote-sensing","title":"An Empirical Study of Remote Sensing Pretraining","arxiv_id":"2204.02825","date":"2022-04-06","proceeding":null,"authors":["Di Wang","Jing Zhang","Bo Du","Gui-Song Xia","DaCheng Tao"],"abstract":"Deep learning has largely reshaped remote sensing (RS) research for aerial image understanding and made a great success. Nevertheless, most of the existing deep models are initialized with the ImageNet pretrained weights. Since natural images inevitably present a large domain gap relative to aerial images, probably limiting the finetuning performance on downstream aerial scene tasks. This issue motivates us to conduct an empirical study of remote sensing pretraining (RSP) on aerial images. To this end, we train different networks from scratch with the help of the largest RS scene recognition dataset up to now -- MillionAID, to obtain a series of RS pretrained backbones, including both convolutional neural networks (CNN) and vision transformers such as Swin and ViTAE, which have shown promising performance on computer vision tasks. Then, we investigate the impact of RSP on representative downstream tasks including scene recognition, semantic segmentation, object detection, and change detection using these CNN and vision transformer backbones. Empirical study shows that RSP can help deliver distinctive performances in scene recognition tasks and in perceiving RS related semantics such as \"Bridge\" and \"Airplane\". We also find that, although RSP mitigates the data discrepancies of traditional ImageNet pretraining on RS images, it may still suffer from task discrepancies, where downstream tasks require different representations from scene recognition tasks. These findings call for further research efforts on both large-scale pretraining datasets and effective pretraining methods. The codes and pretrained models will be released at https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing.","url_abs":"https://arxiv.org/abs/2204.02825v4","url_pdf":"https://arxiv.org/pdf/2204.02825v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-empirical-study-of-remote-sensing","repo_url":"https://github.com/vitae-transformer/rsp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"an-empirical-study-of-remote-sensing","repo_url":"https://github.com/vitae-transformer/vitae-transformer-remote-sensing","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"aerial-scene-classification","task_name":"Aerial Scene Classification"},{"task_slug":"building-change-detection-for-remote-sensing","task_name":"Building change detection for remote sensing images"},{"task_slug":"change-detection","task_name":"Change Detection"},{"task_slug":"change-detection-for-remote-sensing-images","task_name":"Change detection for remote sensing images"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-in-aerial-images","task_name":"Object Detection In Aerial Images"},{"task_slug":"scene-recognition","task_name":"Scene Recognition"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/building-change-detection-for-remote-sensing","task":"Building change detection for remote sensing images","dataset":"LEVIR-CD","model":"IMP-ViTAEv2-S-BIT","rank_in_archive_order":16,"of":37,"metrics":{"F1":"91.26"},"uses_additional_data":true},{"leaderboard":"/sota/building-change-detection-for-remote-sensing","task":"Building change detection for remote sensing images","dataset":"LEVIR-CD","model":"RSP-ViTAEv2-S-BIT","rank_in_archive_order":21,"of":37,"metrics":{"F1":"90.93","IoU":"84.95"},"uses_additional_data":true},{"leaderboard":"/sota/building-change-detection-for-remote-sensing","task":"Building change detection for remote sensing images","dataset":"LEVIR-CD","model":"RSP-ResNet-50","rank_in_archive_order":28,"of":37,"metrics":{"F1":"90.10"},"uses_additional_data":true},{"leaderboard":"/sota/building-change-detection-for-remote-sensing","task":"Building change detection for remote sensing images","dataset":"LEVIR-CD","model":"RSP-Swin-T","rank_in_archive_order":29,"of":37,"metrics":{"F1":"90.10"},"uses_additional_data":true},{"leaderboard":"/sota/change-detection-for-remote-sensing-images-on","task":"Change detection for remote sensing images","dataset":"CDD Dataset (season-varying)","model":"IMP-ViTAEv2-S-BIT","rank_in_archive_order":9,"of":25,"metrics":{"F1-Score":"0.9702"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-for-remote-sensing-images-on","task":"Change detection for remote sensing images","dataset":"CDD Dataset (season-varying)","model":"RSP-ViTAEv2-S-BIT","rank_in_archive_order":11,"of":25,"metrics":{"F1-Score":"0.9681"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-for-remote-sensing-images-on","task":"Change detection for remote sensing images","dataset":"CDD Dataset (season-varying)","model":"RSP-ResNet-50-BIT","rank_in_archive_order":14,"of":25,"metrics":{"F1-Score":"0.96"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-for-remote-sensing-images-on","task":"Change detection for remote sensing images","dataset":"CDD Dataset (season-varying)","model":"RSP-Swin-T-BIT","rank_in_archive_order":17,"of":25,"metrics":{"F1-Score":"0.9521"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dota-1","task":"Object Detection In Aerial Images","dataset":"DOTA","model":"RSP-ViTAEv2-S-FPN-ORCN","rank_in_archive_order":31,"of":58,"metrics":{"mAP":"77.72%"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dota-1","task":"Object Detection In Aerial Images","dataset":"DOTA","model":"IMP-ViTAEv2-S-FPN-ORCN","rank_in_archive_order":34,"of":58,"metrics":{"mAP":"77.38%"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dota-1","task":"Object Detection In Aerial Images","dataset":"DOTA","model":"RSP-ResNet-50-FPN-ORCN","rank_in_archive_order":42,"of":58,"metrics":{"mAP":"76.50%"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dota-1","task":"Object Detection In Aerial Images","dataset":"DOTA","model":"RSP-Swin-T-FPN-ORCN","rank_in_archive_order":44,"of":58,"metrics":{"mAP":"76.12%"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-in-aerial-images-on-hrsc2016","task":"Object Detection In Aerial Images","dataset":"HRSC2016","model":"RSP-ViTAEv2-S-FPN-ORCN","rank_in_archive_order":6,"of":9,"metrics":{"mAP-07":"90.4"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-in-aerial-images-on-hrsc2016","task":"Object Detection In Aerial Images","dataset":"HRSC2016","model":"IMP-ViTAEv2-S-FPN-ORCN","rank_in_archive_order":7,"of":9,"metrics":{"mAP-07":"90.4"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-in-aerial-images-on-hrsc2016","task":"Object Detection In Aerial Images","dataset":"HRSC2016","model":"RSP-ResNet-50-FPN-ORCN","rank_in_archive_order":8,"of":9,"metrics":{"mAP-07":"90.3"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-in-aerial-images-on-hrsc2016","task":"Object Detection In Aerial Images","dataset":"HRSC2016","model":"RSP-Swin-T-FPN-ORCN","rank_in_archive_order":9,"of":9,"metrics":{"mAP-07":"90.0"},"uses_additional_data":true},{"leaderboard":"/sota/semantic-segmentation-on-isprs-potsdam","task":"Semantic Segmentation","dataset":"ISPRS Potsdam","model":"IMP-ViTAEv2-S-UperNet","rank_in_archive_order":7,"of":20,"metrics":{"Overall Accuracy":"91.6"},"uses_additional_data":true},{"leaderboard":"/sota/semantic-segmentation-on-isprs-potsdam","task":"Semantic Segmentation","dataset":"ISPRS Potsdam","model":"RSP-ViTAEv2-S-UperNet","rank_in_archive_order":12,"of":20,"metrics":{"Overall Accuracy":"91.21"},"uses_additional_data":true},{"leaderboard":"/sota/semantic-segmentation-on-isprs-potsdam","task":"Semantic Segmentation","dataset":"ISPRS Potsdam","model":"RSP-Swin-T-UperNet","rank_in_archive_order":14,"of":20,"metrics":{"Overall Accuracy":"90.78"},"uses_additional_data":true},{"leaderboard":"/sota/semantic-segmentation-on-isprs-potsdam","task":"Semantic Segmentation","dataset":"ISPRS Potsdam","model":"RSP-ResNet-50-UperNet","rank_in_archive_order":16,"of":20,"metrics":{"Overall Accuracy":"90.61"},"uses_additional_data":true},{"leaderboard":"/sota/semantic-segmentation-on-isaid","task":"Semantic Segmentation","dataset":"iSAID","model":"IMP-ViTAEv2-S-UperNet","rank_in_archive_order":12,"of":19,"metrics":{"mIoU":"65.3"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-isaid","task":"Semantic Segmentation","dataset":"iSAID","model":"RSP-ViTAEv2-S-UperNet","rank_in_archive_order":15,"of":19,"metrics":{"mIoU":"64.3"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-isaid","task":"Semantic Segmentation","dataset":"iSAID","model":"RSP-Swin-T-UperNet","rank_in_archive_order":16,"of":19,"metrics":{"mIoU":"64.1"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-isaid","task":"Semantic Segmentation","dataset":"iSAID","model":"RSP-ResNet-50-UperNet","rank_in_archive_order":19,"of":19,"metrics":{"mIoU":"61.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.02825","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}