{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mtp-advancing-remote-sensing-foundation-model","title":"MTP: Advancing Remote Sensing Foundation Model via Multi-Task Pretraining","arxiv_id":"2403.13430","date":"2024-03-20","proceeding":null,"authors":["Di Wang","Jing Zhang","Minqiang Xu","Lin Liu","Dongsheng Wang","Erzhong Gao","Chengxi Han","HaoNan Guo","Bo Du","DaCheng Tao","Liangpei Zhang"],"abstract":"Foundation models have reshaped the landscape of Remote Sensing (RS) by enhancing various image interpretation tasks. Pretraining is an active research topic, encompassing supervised and self-supervised learning methods to initialize model weights effectively. However, transferring the pretrained models to downstream tasks may encounter task discrepancy due to their formulation of pretraining as image classification or object discrimination tasks. In this study, we explore the Multi-Task Pretraining (MTP) paradigm for RS foundation models to address this issue. Using a shared encoder and task-specific decoder architecture, we conduct multi-task supervised pretraining on the SAMRS dataset, encompassing semantic segmentation, instance segmentation, and rotated object detection. MTP supports both convolutional neural networks and vision transformer foundation models with over 300 million parameters. The pretrained models are finetuned on various RS downstream tasks, such as scene classification, horizontal and rotated object detection, semantic segmentation, and change detection. Extensive experiments across 14 datasets demonstrate the superiority of our models over existing ones of similar size and their competitive performance compared to larger state-of-the-art models, thus validating the effectiveness of MTP.","url_abs":"https://arxiv.org/abs/2403.13430v2","url_pdf":"https://arxiv.org/pdf/2403.13430v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mtp-advancing-remote-sensing-foundation-model","repo_url":"https://github.com/vitae-transformer/mtp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"mtp-advancing-remote-sensing-foundation-model","repo_url":"https://github.com/cuzyoung/crossearth","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"aerial-scene-classification","task_name":"Aerial Scene Classification"},{"task_slug":"building-change-detection-for-remote-sensing","task_name":"Building change detection for remote sensing images"},{"task_slug":"change-detection","task_name":"Change Detection"},{"task_slug":"change-detection-for-remote-sensing-images","task_name":"Change detection for remote sensing images"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-in-aerial-images","task_name":"Object Detection In Aerial Images"},{"task_slug":"oriented-object-detection","task_name":"Oriented Object Detection"},{"task_slug":"scene-classification","task_name":"Scene Classification"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/building-change-detection-for-remote-sensing","task":"Building change detection for remote sensing images","dataset":"LEVIR-CD","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":1,"of":37,"metrics":{"F1":"92.67","Params(M)":"305"},"uses_additional_data":false},{"leaderboard":"/sota/building-change-detection-for-remote-sensing","task":"Building change detection for remote sensing images","dataset":"LEVIR-CD","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":2,"of":37,"metrics":{"F1":"92.54","Params(M)":"335"},"uses_additional_data":false},{"leaderboard":"/sota/building-change-detection-for-remote-sensing","task":"Building change detection for remote sensing images","dataset":"LEVIR-CD","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":5,"of":37,"metrics":{"F1":"92.22","Params(M)":"86"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-cdd-dataset-season-1","task":"Change Detection","dataset":"CDD Dataset (season-varying)","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":2,"of":18,"metrics":{"F1-Score":"98.33"},"uses_additional_data":true},{"leaderboard":"/sota/change-detection-on-cdd-dataset-season-1","task":"Change Detection","dataset":"CDD Dataset (season-varying)","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":3,"of":18,"metrics":{"F1-Score":"97.98"},"uses_additional_data":true},{"leaderboard":"/sota/change-detection-on-cdd-dataset-season-1","task":"Change Detection","dataset":"CDD Dataset (season-varying)","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":5,"of":18,"metrics":{"F1-Score":"97.87"},"uses_additional_data":true},{"leaderboard":"/sota/change-detection-on-clcd","task":"Change Detection","dataset":"CLCD","model":"MTP (ViT-B + RVSA)","rank_in_archive_order":2,"of":3,"metrics":{"F1":"80.3"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-egy-bcd","task":"Change Detection","dataset":"EGY-BCD","model":"MTP (VIT-B+RVSA)","rank_in_archive_order":1,"of":3,"metrics":{"F1":"85.9"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-gvlm","task":"Change Detection","dataset":"GVLM","model":"MTP (ViT-B + RVSA)","rank_in_archive_order":3,"of":4,"metrics":{"F1":"89.9"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-levir-cd","task":"Change Detection","dataset":"LEVIR-CD","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":2,"of":28,"metrics":{"F1":"92.67"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-levir-cd","task":"Change Detection","dataset":"LEVIR-CD","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":3,"of":28,"metrics":{"F1":"92.54"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-levir-cd","task":"Change Detection","dataset":"LEVIR-CD","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":9,"of":28,"metrics":{"F1":"92.22"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-oscd-3ch","task":"Change Detection","dataset":"OSCD - 3ch","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":1,"of":6,"metrics":{"F1":"55.92"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-oscd-3ch","task":"Change Detection","dataset":"OSCD - 3ch","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":2,"of":6,"metrics":{"F1":"55.61"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-oscd-3ch","task":"Change Detection","dataset":"OSCD - 3ch","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":4,"of":6,"metrics":{"F1":"53.36"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-whu-building-dataset","task":"Change Detection","dataset":"WHU Building Dataset","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":1,"of":7,"metrics":{"F1-score":"0.9559"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-whu-building-dataset","task":"Change Detection","dataset":"WHU Building Dataset","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":3,"of":7,"metrics":{"F1-score":"0.9475"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-on-whu-building-dataset","task":"Change Detection","dataset":"WHU Building Dataset","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":5,"of":7,"metrics":{"F1-score":"0.9432"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-for-remote-sensing-images-on","task":"Change detection for remote sensing images","dataset":"CDD Dataset (season-varying)","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":2,"of":25,"metrics":{"F1-Score":"0.9833"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-for-remote-sensing-images-on","task":"Change detection for remote sensing images","dataset":"CDD Dataset (season-varying)","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":3,"of":25,"metrics":{"F1-Score":"0.9798"},"uses_additional_data":false},{"leaderboard":"/sota/change-detection-for-remote-sensing-images-on","task":"Change detection for remote sensing images","dataset":"CDD Dataset (season-varying)","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":4,"of":25,"metrics":{"F1-Score":"0.9787"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-eurosat","task":"Image Classification","dataset":"EuroSAT","model":"IMP+MTP(IntenImage-XL)","rank_in_archive_order":2,"of":15,"metrics":{"Accuracy (%)":"99.24"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-eurosat","task":"Image Classification","dataset":"EuroSAT","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":9,"of":15,"metrics":{"Accuracy (%)":"98.78"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-eurosat","task":"Image Classification","dataset":"EuroSAT","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":10,"of":15,"metrics":{"Accuracy (%)":"98.76"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dior","task":"Object Detection In Aerial Images","dataset":"DIOR","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":1,"of":4,"metrics":{"AP50":"81.1"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dior","task":"Object Detection In Aerial Images","dataset":"DIOR","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":2,"of":4,"metrics":{"AP50":"79.4"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dior","task":"Object Detection In Aerial Images","dataset":"DIOR","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":3,"of":4,"metrics":{"AP50":"78.0"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dior-r","task":"Object Detection In Aerial Images","dataset":"DIOR-R","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":1,"of":9,"metrics":{"mAP":"74.54"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dior-r","task":"Object Detection In Aerial Images","dataset":"DIOR-R","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":3,"of":9,"metrics":{"mAP":"72.17"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dior-r","task":"Object Detection In Aerial Images","dataset":"DIOR-R","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":4,"of":9,"metrics":{"mAP":"71.29"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dota-1","task":"Object Detection In Aerial Images","dataset":"DOTA","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":9,"of":58,"metrics":{"mAP":"81.66%"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dota-1","task":"Object Detection In Aerial Images","dataset":"DOTA","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":15,"of":58,"metrics":{"mAP":"80.77%"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-in-aerial-images-on-dota-1","task":"Object Detection In Aerial Images","dataset":"DOTA","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":17,"of":58,"metrics":{"mAP":"80.67%"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-in-aerial-images-on-fair1m-2","task":"Object Detection In Aerial Images","dataset":"FAIR1M-2.0","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":1,"of":3,"metrics":{"mAP":"53.00"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-fair1m-2","task":"Object Detection In Aerial Images","dataset":"FAIR1M-2.0","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":2,"of":3,"metrics":{"mAP":"51.92"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-fair1m-2","task":"Object Detection In Aerial Images","dataset":"FAIR1M-2.0","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":3,"of":3,"metrics":{"mAP":"50.93"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-xview","task":"Object Detection In Aerial Images","dataset":"xView","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":1,"of":3,"metrics":{"AP50":"19.4"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-xview","task":"Object Detection In Aerial Images","dataset":"xView","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":2,"of":3,"metrics":{"AP50":"18.2"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-in-aerial-images-on-xview","task":"Object Detection In Aerial Images","dataset":"xView","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":3,"of":3,"metrics":{"AP50":"16.4"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-loveda","task":"Semantic Segmentation","dataset":"LoveDA","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":6,"of":19,"metrics":{"Category mIoU":"54.17"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-loveda","task":"Semantic Segmentation","dataset":"LoveDA","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":7,"of":19,"metrics":{"Category mIoU":"54.17"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-loveda","task":"Semantic Segmentation","dataset":"LoveDA","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":17,"of":19,"metrics":{"Category mIoU":"52.39"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-spacenet-1","task":"Semantic Segmentation","dataset":"SpaceNet 1","model":"MAE+MTP(ViT-L)","rank_in_archive_order":1,"of":10,"metrics":{"Mean IoU":"79.69"},"uses_additional_data":true},{"leaderboard":"/sota/semantic-segmentation-on-spacenet-1","task":"Semantic Segmentation","dataset":"SpaceNet 1","model":"MAE+MTP(ViT-B+RVSA)","rank_in_archive_order":2,"of":10,"metrics":{"Mean IoU":"79.63"},"uses_additional_data":true},{"leaderboard":"/sota/semantic-segmentation-on-spacenet-1","task":"Semantic Segmentation","dataset":"SpaceNet 1","model":"MAE+MTP(ViT-L+RVSA)","rank_in_archive_order":3,"of":10,"metrics":{"Mean IoU":"79.54"},"uses_additional_data":true},{"leaderboard":"/sota/semantic-segmentation-on-spacenet-1","task":"Semantic Segmentation","dataset":"SpaceNet 1","model":"IMP+MTP(InternImage-XL)","rank_in_archive_order":5,"of":10,"metrics":{"Mean IoU":"79.16"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.13430","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.13430"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vitae-transformer/mtp","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cuzyoung/crossearth","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran_draft_wrong":2,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e26853276bb3dd32","entry":"main_process","repo":"vitae-transformer/mtp","repo_kind":"official","path":"Multi-Task_Pretrain/main_pretrain.py","file_url":"https://github.com/vitae-transformer/mtp/blob/HEAD/Multi-Task_Pretrain/main_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e26853276bb3dd32"}},{"code_sha256_prefix":"e7fcb6d9ac9deaf4","entry":"named_apply","repo":"cuzyoung/crossearth","repo_kind":"listed","path":"CrossEarth/models/backbones/dino_v2.py","file_url":"https://github.com/cuzyoung/crossearth/blob/HEAD/CrossEarth/models/backbones/dino_v2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7fcb6d9ac9deaf4"}},{"code_sha256_prefix":"f9fd6241d935f07b","entry":"window_partition","repo":"cuzyoung/crossearth","repo_kind":"listed","path":"CrossEarth/models/backbones/vit_rvsa_mtp.py","file_url":"https://github.com/cuzyoung/crossearth/blob/HEAD/CrossEarth/models/backbones/vit_rvsa_mtp.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9fd6241d935f07b"}},{"code_sha256_prefix":"fb32094c6dbece71","entry":"window_reverse","repo":"cuzyoung/crossearth","repo_kind":"listed","path":"CrossEarth/models/backbones/vit_rvsa_mtp.py","file_url":"https://github.com/cuzyoung/crossearth/blob/HEAD/CrossEarth/models/backbones/vit_rvsa_mtp.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb32094c6dbece71"}},{"code_sha256_prefix":"89b02b1673f5f567","entry":"calc_rel_pos_spatial","repo":"cuzyoung/crossearth","repo_kind":"listed","path":"CrossEarth/models/backbones/vit_rvsa_mtp.py","file_url":"https://github.com/cuzyoung/crossearth/blob/HEAD/CrossEarth/models/backbones/vit_rvsa_mtp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89b02b1673f5f567"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}