{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/omnisat-self-supervised-modality-fusion-for","title":"OmniSat: Self-Supervised Modality Fusion for Earth Observation","arxiv_id":"2404.08351","date":"2024-04-12","proceeding":null,"authors":["Guillaume Astruc","Nicolas Gonthier","Clement Mallet","Loic Landrieu"],"abstract":"The diversity and complementarity of sensors available for Earth Observations (EO) calls for developing bespoke self-supervised multimodal learning approaches. However, current multimodal EO datasets and models typically focus on a single data type, either mono-date images or time series, which limits their impact. To address this issue, we introduce OmniSat, a novel architecture able to merge diverse EO modalities into expressive features without labels by exploiting their alignment. To demonstrate the advantages of our approach, we create two new multimodal datasets by augmenting existing ones with new modalities. As demonstrated for three downstream tasks -- forestry, land cover classification, and crop mapping -- OmniSat can learn rich representations without supervision, leading to state-of-the-art performances in semi- and fully supervised settings. Furthermore, our multimodal pretraining scheme improves performance even when only one modality is available for inference. The code and dataset are available at https://github.com/gastruc/OmniSat.","url_abs":"https://arxiv.org/abs/2404.08351v3","url_pdf":"https://arxiv.org/pdf/2404.08351v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"omnisat-self-supervised-modality-fusion-for","repo_url":"https://github.com/gastruc/omnisat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"earth-observation","task_name":"Earth Observation"},{"task_slug":"land-cover-classification","task_name":"Land Cover Classification"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2404.08351","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.08351"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gastruc/omnisat","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/gastruc/OmniSat","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_fixture":3,"ran_draft_wrong":6,"ran_honours":2,"unverified":9},"by_repo_kind":{"official":{"samples":23,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6f2776fda8eaa86b","entry":"METHOD","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f2776fda8eaa86b"}},{"code_sha256_prefix":"3c90a55d42ebd29a","entry":"_rp_2d_cross_cols","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c90a55d42ebd29a"}},{"code_sha256_prefix":"afa68c96506b7ca4","entry":"_rp_2d_cross_rows","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afa68c96506b7ca4"}},{"code_sha256_prefix":"2afb88f76e6694db","entry":"_rp_2d_euclidean","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2afb88f76e6694db"}},{"code_sha256_prefix":"5e64867591fc57ca","entry":"_rp_2d_product","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e64867591fc57ca"}},{"code_sha256_prefix":"ea730722397f9780","entry":"_rp_2d_quant","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea730722397f9780"}},{"code_sha256_prefix":"ba23bbb042430585","entry":"get_absolute_positions","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba23bbb042430585"}},{"code_sha256_prefix":"ab89b48444865c77","entry":"get_bucket_ids_2d_without_skip","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab89b48444865c77"}},{"code_sha256_prefix":"0deaebd0135ec340","entry":"get_num_buckets","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0deaebd0135ec340"}},{"code_sha256_prefix":"b0cd1557c2946d8d","entry":"get_pylogger","repo":"gastruc/omnisat","repo_kind":"official","path":"src/utils/pylogger.py","file_url":"https://github.com/gastruc/omnisat/blob/HEAD/src/utils/pylogger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0cd1557c2946d8d"}},{"code_sha256_prefix":"8abf0de547843739","entry":"get_rpe_config","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8abf0de547843739"}},{"code_sha256_prefix":"c637a66e6e3977d2","entry":"get_single_rpe_config","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c637a66e6e3977d2"}},{"code_sha256_prefix":"c69948747b50336f","entry":"piecewise_index","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c69948747b50336f"}},{"code_sha256_prefix":"faf9bf7e5bfa997c","entry":"replace_nans_with_mean","repo":"gastruc/omnisat","repo_kind":"official","path":"src/utils/preprocess_tsaits.py","file_url":"https://github.com/gastruc/omnisat/blob/HEAD/src/utils/preprocess_tsaits.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"faf9bf7e5bfa997c"}},{"code_sha256_prefix":"0a1f41cf0a6f4305","entry":"CrossRPEAttention","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a1f41cf0a6f4305"}},{"code_sha256_prefix":"5293629e3ba81145","entry":"CrossRPEBlock","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5293629e3ba81145"}},{"code_sha256_prefix":"817465c178ae5eb3","entry":"OmniModule","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"817465c178ae5eb3"}},{"code_sha256_prefix":"ad51e57d0f69b066","entry":"RPEAttention","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad51e57d0f69b066"}},{"code_sha256_prefix":"47ddca6650a8f8cf","entry":"RPEBlock","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47ddca6650a8f8cf"}},{"code_sha256_prefix":"443aa695f6cd8e56","entry":"build_rpe","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"443aa695f6cd8e56"}},{"code_sha256_prefix":"aa20f508b856826b","entry":"get_bucket_ids_2d","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa20f508b856826b"}},{"code_sha256_prefix":"58768324e3554ec0","entry":"iRPE","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58768324e3554ec0"}},{"code_sha256_prefix":"f7f7cb2bdbfa66d9","entry":"iRPE_Cross","repo":"gastruc/OmniSat","repo_kind":"official","path":"src/models/networks/encoder/Omni.py","file_url":"https://github.com/gastruc/OmniSat/blob/HEAD/src/models/networks/encoder/Omni.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7f7cb2bdbfa66d9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}