{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unveiling-the-potential-of-segment-anything","title":"Unveiling the Potential of Segment Anything Model 2 for RGB-Thermal Semantic Segmentation with Language Guidance","arxiv_id":"2503.02581","date":"2025-03-04","proceeding":null,"authors":["Jiayi Zhao","Fei Teng","Kai Luo","Guoqiang Zhao","Zhiyong Li","Xu Zheng","Kailun Yang"],"abstract":"The perception capability of robotic systems relies on the richness of the dataset. Although Segment Anything Model 2 (SAM2), trained on large datasets, demonstrates strong perception potential in perception tasks, its inherent training paradigm prevents it from being suitable for RGB-T tasks. To address these challenges, we propose SHIFNet, a novel SAM2-driven Hybrid Interaction Paradigm that unlocks the potential of SAM2 with linguistic guidance for efficient RGB-Thermal perception. Our framework consists of two key components: (1) Semantic-Aware Cross-modal Fusion (SACF) module that dynamically balances modality contributions through text-guided affinity learning, overcoming SAM2's inherent RGB bias; (2) Heterogeneous Prompting Decoder (HPD) that enhances global semantic information through a semantic enhancement module and then combined with category embeddings to amplify cross-modal semantic consistency. With 32.27M trainable parameters, SHIFNet achieves state-of-the-art segmentation performance on public benchmarks, reaching 89.8% on PST900 and 67.8% on FMB, respectively. The framework facilitates the adaptation of pre-trained large models to RGB-T segmentation tasks, effectively mitigating the high costs associated with data collection while endowing robotic systems with comprehensive perception capabilities. The source code will be made publicly available at https://github.com/iAsakiT3T/SHIFNet.","url_abs":"https://arxiv.org/abs/2503.02581v1","url_pdf":"https://arxiv.org/pdf/2503.02581v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unveiling-the-potential-of-segment-anything","repo_url":"https://github.com/iasakit3t/shifnet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"thermal-image-segmentation","task_name":"Thermal Image Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-segmentation-on-fmb-dataset","task":"Semantic Segmentation","dataset":"FMB Dataset","model":"SHIFNet (RGB-Infrared)","rank_in_archive_order":2,"of":14,"metrics":{"mIoU":"67.8"},"uses_additional_data":false},{"leaderboard":"/sota/thermal-image-segmentation-on-mfn-dataset","task":"Thermal Image Segmentation","dataset":"MFN Dataset","model":"SHIFNet","rank_in_archive_order":10,"of":55,"metrics":{"mIOU":"59.2"},"uses_additional_data":false},{"leaderboard":"/sota/thermal-image-segmentation-on-pst900","task":"Thermal Image Segmentation","dataset":"PST900","model":"SHIFNet","rank_in_archive_order":1,"of":22,"metrics":{"mIoU":"89.8"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.02581","atlas_url":"https://app.syntology.ai/?focus=2503.02581","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}