{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-remote-sensing-with-harmonized","title":"Efficient Remote Sensing with Harmonized Transfer Learning and Modality Alignment","arxiv_id":"2404.18253","date":"2024-04-28","proceeding":null,"authors":["Tengjun Huang"],"abstract":"With the rise of Visual and Language Pretraining (VLP), an increasing number of downstream tasks are adopting the paradigm of pretraining followed by fine-tuning. Although this paradigm has demonstrated potential in various multimodal downstream tasks, its implementation in the remote sensing domain encounters some obstacles. Specifically, the tendency for same-modality embeddings to cluster together impedes efficient transfer learning. To tackle this issue, we review the aim of multimodal transfer learning for downstream tasks from a unified perspective, and rethink the optimization process based on three distinct objectives. We propose \"Harmonized Transfer Learning and Modality Alignment (HarMA)\", a method that simultaneously satisfies task constraints, modality alignment, and single-modality uniform alignment, while minimizing training overhead through parameter-efficient fine-tuning. Remarkably, without the need for external data for training, HarMA achieves state-of-the-art performance in two popular multimodal retrieval tasks in the field of remote sensing. Our experiments reveal that HarMA achieves competitive and even superior performance to fully fine-tuned models with only minimal adjustable parameters. Due to its simplicity, HarMA can be integrated into almost all existing multimodal pretraining models. We hope this method can facilitate the efficient application of large models to a wide range of downstream tasks while significantly reducing the resource consumption. Code is available at https://github.com/seekerhuang/HarMA.","url_abs":"https://arxiv.org/abs/2404.18253v5","url_pdf":"https://arxiv.org/pdf/2404.18253v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-remote-sensing-with-harmonized","repo_url":"https://github.com/seekerhuang/harma","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"image-to-text-retrieval","task_name":"Image-to-Text Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/cross-modal-retrieval-on-rsicd","task":"Cross-Modal Retrieval","dataset":"RSICD","model":"HarMA (w/ GeoRSCLIP)","rank_in_archive_order":1,"of":10,"metrics":{"Image-to-text R@1":"20.52%","Mean Recall":"38.95%","text-to-image R@1":"15.84%"},"uses_additional_data":false},{"leaderboard":"/sota/cross-modal-retrieval-on-rsitmd","task":"Cross-Modal Retrieval","dataset":"RSITMD","model":"HarMA (w/ GeoRSCLIP)","rank_in_archive_order":1,"of":10,"metrics":{"Image-to-text R@1":"32.74%","Mean Recall":"52.27%","text-to-imageR@1":"25.62%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.18253","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.18253"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/seekerhuang/harma","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e1cf6464da33ec0c","entry":"read_json","repo":"seekerhuang/harma","repo_kind":"official","path":"models/model_retrieval.py","file_url":"https://github.com/seekerhuang/harma/blob/HEAD/models/model_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1cf6464da33ec0c"}},{"code_sha256_prefix":"3d8d0c7edd3e4998","entry":"AllGather","repo":"seekerhuang/harma","repo_kind":"official","path":"models/model_retrieval.py","file_url":"https://github.com/seekerhuang/harma/blob/HEAD/models/model_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3d8d0c7edd3e4998"}},{"code_sha256_prefix":"b98dc0a1fd4cd372","entry":"HarMA","repo":"seekerhuang/harma","repo_kind":"official","path":"models/model_retrieval.py","file_url":"https://github.com/seekerhuang/harma/blob/HEAD/models/model_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b98dc0a1fd4cd372"}},{"code_sha256_prefix":"71b3b1acfac0847d","entry":"HarMABase","repo":"seekerhuang/harma","repo_kind":"official","path":"models/model_retrieval.py","file_url":"https://github.com/seekerhuang/harma/blob/HEAD/models/model_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"71b3b1acfac0847d"}},{"code_sha256_prefix":"bc9c8b1db97a5749","entry":"interpolate_relative_pos_embed","repo":"seekerhuang/harma","repo_kind":"official","path":"models/model_retrieval.py","file_url":"https://github.com/seekerhuang/harma/blob/HEAD/models/model_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bc9c8b1db97a5749"}},{"code_sha256_prefix":"a0d20fab53560545","entry":"load_pretrained_harma","repo":"seekerhuang/harma","repo_kind":"official","path":"models/model_retrieval.py","file_url":"https://github.com/seekerhuang/harma/blob/HEAD/models/model_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a0d20fab53560545"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}