{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sigma-siamese-mamba-network-for-multi-modal","title":"Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation","arxiv_id":"2404.04256","date":"2024-04-05","proceeding":null,"authors":["Zifu Wan","Pingping Zhang","Yuhao Wang","Silong Yong","Simon Stepputtis","Katia Sycara","Yaqi Xie"],"abstract":"Multi-modal semantic segmentation significantly enhances AI agents' perception and scene understanding, especially under adverse conditions like low-light or overexposed environments. Leveraging additional modalities (X-modality) like thermal and depth alongside traditional RGB provides complementary information, enabling more robust and reliable prediction. In this work, we introduce Sigma, a Siamese Mamba network for multi-modal semantic segmentation utilizing the advanced Mamba. Unlike conventional methods that rely on CNNs, with their limited local receptive fields, or Vision Transformers (ViTs), which offer global receptive fields at the cost of quadratic complexity, our model achieves global receptive fields with linear complexity. By employing a Siamese encoder and innovating a Mamba-based fusion mechanism, we effectively select essential information from different modalities. A decoder is then developed to enhance the channel-wise modeling ability of the model. Our proposed method is rigorously evaluated on both RGB-Thermal and RGB-Depth semantic segmentation tasks, demonstrating its superiority and marking the first successful application of State Space Models (SSMs) in multi-modal perception tasks. Code is available at https://github.com/zifuwan/Sigma.","url_abs":"https://arxiv.org/abs/2404.04256v2","url_pdf":"https://arxiv.org/pdf/2404.04256v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sigma-siamese-mamba-network-for-multi-modal","repo_url":"https://github.com/zifuwan/sigma","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"state-space-models","task_name":"State Space Models"},{"task_slug":"thermal-image-segmentation","task_name":"Thermal Image Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/thermal-image-segmentation-on-mfn-dataset","task":"Thermal Image Segmentation","dataset":"MFN Dataset","model":"Sigma-base","rank_in_archive_order":4,"of":55,"metrics":{"mIOU":"61.3"},"uses_additional_data":false},{"leaderboard":"/sota/thermal-image-segmentation-on-pst900","task":"Thermal Image Segmentation","dataset":"PST900","model":"Sigma-small","rank_in_archive_order":4,"of":22,"metrics":{"mIoU":"87.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.04256","atlas_url":"https://app.syntology.ai/?focus=2404.04256","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.04256"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zifuwan/sigma","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"83ca13ec3eefbcd1","entry":"random_mirror","repo":"zifuwan/sigma","repo_kind":"official","path":"dataloader/dataloader.py","file_url":"https://github.com/zifuwan/sigma/blob/HEAD/dataloader/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83ca13ec3eefbcd1"}},{"code_sha256_prefix":"3e2c98e28bc11bd4","entry":"random_scale","repo":"zifuwan/sigma","repo_kind":"official","path":"dataloader/dataloader.py","file_url":"https://github.com/zifuwan/sigma/blob/HEAD/dataloader/dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e2c98e28bc11bd4"}},{"code_sha256_prefix":"9a3adc8e6de35ac1","entry":"flops_selective_scan_fn","repo":"zifuwan/sigma","repo_kind":"official","path":"models/builder.py","file_url":"https://github.com/zifuwan/sigma/blob/HEAD/models/builder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a3adc8e6de35ac1"}},{"code_sha256_prefix":"1a7a2a1b97877257","entry":"selective_scan_flop_jit","repo":"zifuwan/sigma","repo_kind":"official","path":"models/builder.py","file_url":"https://github.com/zifuwan/sigma/blob/HEAD/models/builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a7a2a1b97877257"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}