{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ss-mae-spatial-spectral-masked-auto-encoder","title":"SS-MAE: Spatial-Spectral Masked Auto-Encoder for Multi-Source Remote Sensing Image Classification","arxiv_id":"2311.04442","date":"2023-11-08","proceeding":null,"authors":["Junyan Lin","Feng Gao","Xiaocheng Shi","Junyu Dong","Qian Du"],"abstract":"Masked image modeling (MIM) is a highly popular and effective self-supervised learning method for image understanding. Existing MIM-based methods mostly focus on spatial feature modeling, neglecting spectral feature modeling. Meanwhile, existing MIM-based methods use Transformer for feature extraction, some local or high-frequency information may get lost. To this end, we propose a spatial-spectral masked auto-encoder (SS-MAE) for HSI and LiDAR/SAR data joint classification. Specifically, SS-MAE consists of a spatial-wise branch and a spectral-wise branch. The spatial-wise branch masks random patches and reconstructs missing pixels, while the spectral-wise branch masks random spectral channels and reconstructs missing channels. Our SS-MAE fully exploits the spatial and spectral representations of the input data. Furthermore, to complement local features in the training stage, we add two lightweight CNNs for feature extraction. Both global and local features are taken into account for feature modeling. To demonstrate the effectiveness of the proposed SS-MAE, we conduct extensive experiments on three publicly available datasets. Extensive experiments on three multi-source datasets verify the superiority of our SS-MAE compared with several state-of-the-art baselines. The source codes are available at \\url{https://github.com/summitgao/SS-MAE}.","url_abs":"https://arxiv.org/abs/2311.04442v1","url_pdf":"https://arxiv.org/pdf/2311.04442v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ss-mae-spatial-spectral-masked-auto-encoder","repo_url":"https://github.com/summitgao/ss-mae","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"remote-sensing-image-classification","task_name":"Remote Sensing Image Classification"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.04442","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.04442"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/summitgao/ss-mae","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"ran":6,"unverified":7},"by_repo_kind":{"official":{"samples":15,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"c58578dcfe39d5e3","entry":"adapt_input_conv","repo":"summitgao/ss-mae","repo_kind":"official","path":"net/VIT/layers/helpers.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/net/VIT/layers/helpers.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c58578dcfe39d5e3"}},{"code_sha256_prefix":"ae4dd5ff9771ecb5","entry":"build_mask_chan","repo":"summitgao/ss-mae","repo_kind":"official","path":"loss/mae_loss.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/loss/mae_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae4dd5ff9771ecb5"}},{"code_sha256_prefix":"b85eccfd0788b45b","entry":"build_mask_spa","repo":"summitgao/ss-mae","repo_kind":"official","path":"loss/mae_loss.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/loss/mae_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b85eccfd0788b45b"}},{"code_sha256_prefix":"31695defbde45622","entry":"build_optimizer","repo":"summitgao/ss-mae","repo_kind":"official","path":"utils/optimizer_step.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/utils/optimizer_step.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31695defbde45622"}},{"code_sha256_prefix":"39eace7e2822504f","entry":"drop_path","repo":"summitgao/ss-mae","repo_kind":"official","path":"net/VIT/layers/drop.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/net/VIT/layers/drop.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39eace7e2822504f"}},{"code_sha256_prefix":"6dc13e8f8b2c3d96","entry":"load_state_dict","repo":"summitgao/ss-mae","repo_kind":"official","path":"net/VIT/layers/helpers.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/net/VIT/layers/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6dc13e8f8b2c3d96"}},{"code_sha256_prefix":"b33222a09fc93bec","entry":"set_weight_decay","repo":"summitgao/ss-mae","repo_kind":"official","path":"utils/optimizer_step.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/utils/optimizer_step.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b33222a09fc93bec"}},{"code_sha256_prefix":"89421f54d0df827b","entry":"to_2tuple","repo":"summitgao/ss-mae","repo_kind":"official","path":"net/VIT/layers/mlp.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/net/VIT/layers/mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89421f54d0df827b"}},{"code_sha256_prefix":"77e17928d8e92b17","entry":"accuracy","repo":"summitgao/ss-mae","repo_kind":"official","path":"utility.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/utility.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77e17928d8e92b17"}},{"code_sha256_prefix":"5637b67ff7832281","entry":"cal_results","repo":"summitgao/ss-mae","repo_kind":"official","path":"utility.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/utility.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5637b67ff7832281"}},{"code_sha256_prefix":"b0a5beb34716d5a6","entry":"check_keywords_in_name","repo":"summitgao/ss-mae","repo_kind":"official","path":"utils/optimizer_step.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/utils/optimizer_step.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b0a5beb34716d5a6"}},{"code_sha256_prefix":"653d5a3aa2c473be","entry":"drop_block_2d","repo":"summitgao/ss-mae","repo_kind":"official","path":"net/VIT/layers/drop.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/net/VIT/layers/drop.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"653d5a3aa2c473be"}},{"code_sha256_prefix":"97aa7c942cfb7d3d","entry":"drop_block_fast_2d","repo":"summitgao/ss-mae","repo_kind":"official","path":"net/VIT/layers/drop.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/net/VIT/layers/drop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"97aa7c942cfb7d3d"}},{"code_sha256_prefix":"dfc15802c2295917","entry":"output_metric","repo":"summitgao/ss-mae","repo_kind":"official","path":"utility.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/utility.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfc15802c2295917"}},{"code_sha256_prefix":"102b75f3481035d1","entry":"resume_checkpoint","repo":"summitgao/ss-mae","repo_kind":"official","path":"net/VIT/layers/helpers.py","file_url":"https://github.com/summitgao/ss-mae/blob/HEAD/net/VIT/layers/helpers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"102b75f3481035d1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}