{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/surgical-sam-2-real-time-segment-anything-in","title":"Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning","arxiv_id":"2408.07931","date":"2024-08-15","proceeding":null,"authors":["Haofeng Liu","Erli Zhang","Junde Wu","Mingxuan Hong","Yueming Jin"],"abstract":"Surgical video segmentation is a critical task in computer-assisted surgery and is vital for enhancing surgical quality and patient outcomes. Recently, the Segment Anything Model 2 (SAM2) framework has shown superior advancements in image and video segmentation. However, SAM2 struggles with efficiency due to the high computational demands of processing high-resolution images and complex and long-range temporal dynamics in surgical videos. To address these challenges, we introduce Surgical SAM 2 (SurgSAM2), an advanced model to utilize SAM2 with an Efficient Frame Pruning (EFP) mechanism, to facilitate real-time surgical video segmentation. The EFP mechanism dynamically manages the memory bank by selectively retaining only the most informative frames, reducing memory usage and computational cost while maintaining high segmentation accuracy. Our extensive experiments demonstrate that SurgSAM2 significantly improves both efficiency and segmentation accuracy compared to the vanilla SAM2. Remarkably, SurgSAM2 achieves a 3$\\times$ FPS compared with SAM2, while also delivering state-of-the-art performance after fine-tuning with lower-resolution data. These advancements establish SurgSAM2 as a leading model for surgical video analysis, making real-time surgical video segmentation in resource-constrained environments a reality. Our source code is available at https://github.com/jinlab-imvr/Surgical-SAM-2.","url_abs":"https://arxiv.org/abs/2408.07931v2","url_pdf":"https://arxiv.org/pdf/2408.07931v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"surgical-sam-2-real-time-segment-anything-in","repo_url":"https://github.com/jinlab-imvr/surgical-sam-2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"video-segmentation","task_name":"Video Segmentation"},{"task_slug":"video-semantic-segmentation","task_name":"Video Semantic Segmentation"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.07931","atlas_url":"https://app.syntology.ai/?focus=2408.07931","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.07931"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jinlab-imvr/Surgical-SAM-2","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jinlab-imvr/surgical-sam-2","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_violates":1,"ran_draft_wrong":4,"ran_honours":2,"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":16,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bbbfffe7be426cf5","entry":"build_sam2","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"sam2/build_sam.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/sam2/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bbbfffe7be426cf5"}},{"code_sha256_prefix":"7ceaad42da42edd5","entry":"compute_axial_cis","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7ceaad42da42edd5"}},{"code_sha256_prefix":"a6c3974188a5ffb0","entry":"get_1d_sine_pe","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a6c3974188a5ffb0"}},{"code_sha256_prefix":"259650ecd92aae93","entry":"get_per_obj_mask","repo":"jinlab-imvr/Surgical-SAM-2","repo_kind":"official","path":"tools/vos_inference.py","file_url":"https://github.com/jinlab-imvr/Surgical-SAM-2/blob/HEAD/tools/vos_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"259650ecd92aae93"}},{"code_sha256_prefix":"bb060d29525f90eb","entry":"init_t_xy","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb060d29525f90eb"}},{"code_sha256_prefix":"fecc11868aafba36","entry":"load_ann_png","repo":"jinlab-imvr/Surgical-SAM-2","repo_kind":"official","path":"tools/vos_inference.py","file_url":"https://github.com/jinlab-imvr/Surgical-SAM-2/blob/HEAD/tools/vos_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fecc11868aafba36"}},{"code_sha256_prefix":"2e947f9cfdf99dc6","entry":"put_per_obj_mask","repo":"jinlab-imvr/Surgical-SAM-2","repo_kind":"official","path":"tools/vos_inference.py","file_url":"https://github.com/jinlab-imvr/Surgical-SAM-2/blob/HEAD/tools/vos_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e947f9cfdf99dc6"}},{"code_sha256_prefix":"f8f09f78b474ee7c","entry":"reshape_for_broadcast","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8f09f78b474ee7c"}},{"code_sha256_prefix":"4ee20f090e58c425","entry":"select_closest_cond_frames","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4ee20f090e58c425"}},{"code_sha256_prefix":"b532a4eee5754aee","entry":"build_sam2_hf","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"sam2/build_sam.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/sam2/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b532a4eee5754aee"}},{"code_sha256_prefix":"4a1bbe17897a3ec2","entry":"build_sam2_video_predictor","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"sam2/build_sam.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/sam2/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a1bbe17897a3ec2"}},{"code_sha256_prefix":"e473e66fedc883c7","entry":"dice_loss","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"training/loss_fns.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/training/loss_fns.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e473e66fedc883c7"}},{"code_sha256_prefix":"f2e72d81f39d65cd","entry":"get_activation_fn","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f2e72d81f39d65cd"}},{"code_sha256_prefix":"b8f501ca29e77f1a","entry":"iou_loss","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"training/loss_fns.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/training/loss_fns.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8f501ca29e77f1a"}},{"code_sha256_prefix":"6b7c7753844ee197","entry":"sdp_kernel_context","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"sam2/modeling/sam/transformer.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/sam2/modeling/sam/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6b7c7753844ee197"}},{"code_sha256_prefix":"42fec6fdc1faf6f1","entry":"sigmoid_focal_loss","repo":"jinlab-imvr/surgical-sam-2","repo_kind":"official","path":"training/loss_fns.py","file_url":"https://github.com/jinlab-imvr/surgical-sam-2/blob/HEAD/training/loss_fns.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42fec6fdc1faf6f1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}