{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-plasticity-inspired-foundation-model","title":"Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation","arxiv_id":"2403.15356","date":"2024-03-22","proceeding":null,"authors":["Zhitong Xiong","Yi Wang","Fahong Zhang","Adam J. Stewart","Joëlle Hanna","Damian Borth","Ioannis Papoutsis","Bertrand Le Saux","Gustau Camps-Valls","Xiao Xiang Zhu"],"abstract":"The development of foundation models has revolutionized our ability to interpret the Earth's surface using satellite observational data. Traditional models have been siloed, tailored to specific sensors or data types like optical, radar, and hyperspectral, each with its own unique characteristics. This specialization hinders the potential for a holistic analysis that could benefit from the combined strengths of these diverse data sources. Our novel approach introduces the Dynamic One-For-All (DOFA) model, leveraging the concept of neural plasticity in brain science to integrate various data modalities into a single framework adaptively. This dynamic hypernetwork, adjusting to different wavelengths, enables a single versatile Transformer jointly trained on data from five sensors to excel across 12 distinct Earth observation tasks, including sensors never seen during pretraining. DOFA's innovative design offers a promising leap towards more accurate, efficient, and unified Earth observation analysis, showcasing remarkable adaptability and performance in harnessing the potential of multimodal Earth observation data.","url_abs":"https://arxiv.org/abs/2403.15356v2","url_pdf":"https://arxiv.org/pdf/2403.15356v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-plasticity-inspired-foundation-model","repo_url":"https://github.com/zhu-xlab/dofa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"earth-observation","task_name":"Earth Observation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.15356","atlas_url":"https://app.syntology.ai/?focus=2403.15356","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.15356"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhu-xlab/dofa","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a1966f7ff23ec4ef","entry":"get_1d_sincos_pos_embed_from_grid_torch","repo":"zhu-xlab/dofa","repo_kind":"official","path":"wave_dynamic_layer.py","file_url":"https://github.com/zhu-xlab/dofa/blob/HEAD/wave_dynamic_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1966f7ff23ec4ef"}},{"code_sha256_prefix":"1d01ab753c8dc375","entry":"resolve_config","repo":"zhu-xlab/dofa","repo_kind":"official","path":"downstream_tasks/geobench_segmentation/main_finetune.py","file_url":"https://github.com/zhu-xlab/dofa/blob/HEAD/downstream_tasks/geobench_segmentation/main_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d01ab753c8dc375"}},{"code_sha256_prefix":"d7a6855303cdfd94","entry":"resolve_tasks","repo":"zhu-xlab/dofa","repo_kind":"official","path":"downstream_tasks/geobench_segmentation/main_finetune.py","file_url":"https://github.com/zhu-xlab/dofa/blob/HEAD/downstream_tasks/geobench_segmentation/main_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7a6855303cdfd94"}},{"code_sha256_prefix":"db920d6c09aa3985","entry":"resolve_tasks","repo":"zhu-xlab/dofa","repo_kind":"official","path":"downstream_tasks/geobench_classification/main_linear_probe.py","file_url":"https://github.com/zhu-xlab/dofa/blob/HEAD/downstream_tasks/geobench_classification/main_linear_probe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db920d6c09aa3985"}},{"code_sha256_prefix":"017b8166c39169f6","entry":"vit_base_dofa","repo":"zhu-xlab/dofa","repo_kind":"official","path":"hubconf.py","file_url":"https://github.com/zhu-xlab/dofa/blob/HEAD/hubconf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"017b8166c39169f6"}},{"code_sha256_prefix":"33a50be22c1f0de8","entry":"create_dataset","repo":"zhu-xlab/dofa","repo_kind":"official","path":"downstream_tasks/geobench_segmentation/main_finetune.py","file_url":"https://github.com/zhu-xlab/dofa/blob/HEAD/downstream_tasks/geobench_segmentation/main_finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33a50be22c1f0de8"}},{"code_sha256_prefix":"d1713f477bc64e3e","entry":"create_dataset","repo":"zhu-xlab/dofa","repo_kind":"official","path":"downstream_tasks/geobench_classification/main_linear_probe.py","file_url":"https://github.com/zhu-xlab/dofa/blob/HEAD/downstream_tasks/geobench_classification/main_linear_probe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1713f477bc64e3e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}