{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rap-sam-towards-real-time-all-purpose-segment","title":"RAP-SAM: Towards Real-Time All-Purpose Segment Anything","arxiv_id":"2401.10228","date":"2024-01-18","proceeding":null,"authors":["Shilin Xu","Haobo Yuan","Qingyu Shi","Lu Qi","Jingbo Wang","Yibo Yang","Yining Li","Kai Chen","Yunhai Tong","Bernard Ghanem","Xiangtai Li","Ming-Hsuan Yang"],"abstract":"Advanced by transformer architecture, vision foundation models (VFMs) achieve remarkable progress in performance and generalization ability. Segment Anything Model (SAM) is one remarkable model that can achieve generalized segmentation. However, most VFMs cannot run in realtime, which makes it difficult to transfer them into several products. On the other hand, current real-time segmentation mainly has one purpose, such as semantic segmentation on the driving scene. We argue that diverse outputs are needed for real applications. Thus, this work explores a new real-time segmentation setting, named all-purpose segmentation in real-time, to transfer VFMs in real-time deployment. It contains three different tasks, including interactive segmentation, panoptic segmentation, and video segmentation. We aim to use one model to achieve the above tasks in real-time. We first benchmark several strong baselines. Then, we present Real-Time All Purpose SAM (RAP-SAM). It contains an efficient encoder and an efficient decoupled decoder to perform prompt-driven decoding. Moreover, we further explore different training strategies and tuning methods to boost co-training performance further. Our code and model are available at https://github.com/xushilin1/RAP-SAM/.","url_abs":"https://arxiv.org/abs/2401.10228v1","url_pdf":"https://arxiv.org/pdf/2401.10228v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rap-sam-towards-real-time-all-purpose-segment","repo_url":"https://github.com/xushilin1/rap-sam","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"interactive-segmentation","task_name":"Interactive Segmentation"},{"task_slug":"panoptic-segmentation","task_name":"Panoptic Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"video-segmentation","task_name":"Video Segmentation"},{"task_slug":"video-semantic-segmentation","task_name":"Video Semantic Segmentation"}],"methods":[{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.10228","atlas_url":"https://app.syntology.ai/?focus=2401.10228","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.10228"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/xushilin1/RAP-SAM","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xushilin1/rap-sam","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"733d7f0bedcb74c2","entry":"get_rel_pos","repo":"xushilin1/RAP-SAM","repo_kind":"official","path":"ext/sam/image_encoder.py","file_url":"https://github.com/xushilin1/RAP-SAM/blob/HEAD/ext/sam/image_encoder.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"733d7f0bedcb74c2"}},{"code_sha256_prefix":"105fa08885dc36cc","entry":"window_partition","repo":"xushilin1/RAP-SAM","repo_kind":"official","path":"ext/sam/image_encoder.py","file_url":"https://github.com/xushilin1/RAP-SAM/blob/HEAD/ext/sam/image_encoder.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"105fa08885dc36cc"}},{"code_sha256_prefix":"27be441cc8213e52","entry":"window_unpartition","repo":"xushilin1/RAP-SAM","repo_kind":"official","path":"ext/sam/image_encoder.py","file_url":"https://github.com/xushilin1/RAP-SAM/blob/HEAD/ext/sam/image_encoder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27be441cc8213e52"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}