Papers › MemorySAM: Memorize Modalities and Semantics with Segment Anything Model 2 for...
MemorySAM: Memorize Modalities and Semantics with Segment Anything Model 2 for Multi-modal Semantic Segmentation
Chenfei Liao, Xu Zheng, Yuanhuiyi Lyu, Haiwei Xue, Yihong Cao, Jiawen Wang, Kailun Yang, Xuming Hu
Research has focused on Multi-Modal Semantic Segmentation (MMSS), where pixel-wise predictions are derived from multiple visual modalities captured by diverse sensors. Recently, the large vision model, Segment Anything Model 2 (SAM2), has shown strong zero-shot segmentation performance on both images and videos. When extending SAM2 to MMSS, two issues arise: 1. How can SAM2 be adapted to multi-modal data? 2. How can SAM2 better understand semantics? Inspired by cross-frame correlation in videos, we propose to treat multi-modal data as a sequence of frames representing the same scene. Our key idea is to ''memorize'' the modality-agnostic information and 'memorize' the semantics related to the targeted scene. To achieve this, we apply SAM2's memory mechanisms across multi-modal data to capture modality-agnostic features. Meanwhile, to memorize the semantic knowledge, we propose a training-only Semantic Prototype Memory Module (SPMM) to store category-level prototypes across training for facilitating SAM2's transition from instance to semantic segmentation. A prototypical adaptation loss is imposed between global and local prototypes iteratively to align and refine SAM2's semantic understanding. Extensive experimental results demonstrate that our proposed MemorySAM outperforms SoTA methods by large margins on both synthetic and real-world benchmarks (65.38% on DELIVER, 52.88% on MCubeS). Source code will be made publicly available.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Semantic Segmentation | DeLiVER | MemorySAM-B+(R-D-E-L) | mIoU | 65.38 | #8 of 26 | Archive leaderboard | report |
| Semantic Segmentation | DeLiVER | MemorySAM-B+(R-D) | mIoU | 63.48 | #9 of 26 | Archive leaderboard | report |
| Semantic Segmentation | DeLiVER | MemorySAM-B+(R-D-E) | mIoU | 62.42 | #11 of 26 | Archive leaderboard | report |
| Semantic Segmentation | DeLiVER | MemorySAM-B+(RGB) | mIoU | 53.22 | #17 of 26 | Archive leaderboard | report |
| Semantic Segmentation | MCubeS | MemorySAM-B+(RGB-A-D-N) | mIoU | 52.88 | #5 of 22 | Archive leaderboard | report |
| Semantic Segmentation | MCubeS | MemorySAM-B+(RGB-A-D) | mIoU | 52.20 | #8 of 22 | Archive leaderboard | report |
| Semantic Segmentation | MCubeS | MemorySAM-B+(RGB-A) | mIoU | 51.20 | #12 of 22 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections