{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sam2long-enhancing-sam-2-for-long-video","title":"SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree","arxiv_id":"2410.16268","date":"2024-10-21","proceeding":null,"authors":["Shuangrui Ding","Rui Qian","Xiaoyi Dong","Pan Zhang","Yuhang Zang","Yuhang Cao","Yuwei Guo","Dahua Lin","Jiaqi Wang"],"abstract":"The Segment Anything Model 2 (SAM 2) has emerged as a powerful foundation model for object segmentation in both images and videos, paving the way for various downstream video applications. The crucial design of SAM 2 for video segmentation is its memory module, which prompts object-aware memories from previous frames for current frame prediction. However, its greedy-selection memory design suffers from the \"error accumulation\" problem, where an errored or missed mask will cascade and influence the segmentation of the subsequent frames, which limits the performance of SAM 2 toward complex long-term videos. To this end, we introduce SAM2Long, an improved training-free video object segmentation strategy, which considers the segmentation uncertainty within each frame and chooses the video-level optimal results from multiple segmentation pathways in a constrained tree search manner. In practice, we maintain a fixed number of segmentation pathways throughout the video. For each frame, multiple masks are proposed based on the existing pathways, creating various candidate branches. We then select the same fixed number of branches with higher cumulative scores as the new pathways for the next frame. After processing the final frame, the pathway with the highest cumulative score is chosen as the final segmentation result. Benefiting from its heuristic search design, SAM2Long is robust toward occlusions and object reappearances, and can effectively segment and track objects for complex long-term videos. Notably, SAM2Long achieves an average improvement of 3.0 points across all 24 head-to-head comparisons, with gains of up to 5.3 points in J&F on long-term video object segmentation benchmarks such as SA-V and LVOS. The code is released at https://github.com/Mark12Ding/SAM2Long.","url_abs":"https://arxiv.org/abs/2410.16268v2","url_pdf":"https://arxiv.org/pdf/2410.16268v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sam2long-enhancing-sam-2-for-long-video","repo_url":"https://github.com/mark12ding/sam2long","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"heuristic-search","task_name":"Heuristic Search"},{"task_slug":"object","task_name":"Object"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"video-object-segmentation","task_name":"Video Object Segmentation"},{"task_slug":"video-segmentation","task_name":"Video Segmentation"},{"task_slug":"video-semantic-segmentation","task_name":"Video Semantic Segmentation"},{"task_slug":"visual-object-tracking","task_name":"Visual Object Tracking"}],"methods":[{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-object-tracking-on-didi","task":"Visual Object Tracking","dataset":"DiDi","model":"SAM2.1Long","rank_in_archive_order":4,"of":11,"metrics":{"Tracking quality":"0.646"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.16268","atlas_url":"https://app.syntology.ai/?focus=2410.16268","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.16268"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mark12ding/sam2long","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_violates":1,"ran_draft_wrong":2,"ran_honours":1,"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"7ceaad42da42edd5","entry":"compute_axial_cis","repo":"mark12ding/sam2long","repo_kind":"official","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/mark12ding/sam2long/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ceaad42da42edd5"}},{"code_sha256_prefix":"a6c3974188a5ffb0","entry":"get_1d_sine_pe","repo":"mark12ding/sam2long","repo_kind":"official","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/mark12ding/sam2long/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a6c3974188a5ffb0"}},{"code_sha256_prefix":"bb060d29525f90eb","entry":"init_t_xy","repo":"mark12ding/sam2long","repo_kind":"official","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/mark12ding/sam2long/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb060d29525f90eb"}},{"code_sha256_prefix":"f8f09f78b474ee7c","entry":"reshape_for_broadcast","repo":"mark12ding/sam2long","repo_kind":"official","path":"sam2/modeling/position_encoding.py","file_url":"https://github.com/mark12ding/sam2long/blob/HEAD/sam2/modeling/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8f09f78b474ee7c"}},{"code_sha256_prefix":"4ee20f090e58c425","entry":"select_closest_cond_frames","repo":"mark12ding/sam2long","repo_kind":"official","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/mark12ding/sam2long/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ee20f090e58c425"}},{"code_sha256_prefix":"f9850c1d4acdbb5a","entry":"build_sam2","repo":"mark12ding/sam2long","repo_kind":"official","path":"sam2/build_sam.py","file_url":"https://github.com/mark12ding/sam2long/blob/HEAD/sam2/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9850c1d4acdbb5a"}},{"code_sha256_prefix":"de8c2a625f263855","entry":"build_sam2_hf","repo":"mark12ding/sam2long","repo_kind":"official","path":"sam2/build_sam.py","file_url":"https://github.com/mark12ding/sam2long/blob/HEAD/sam2/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de8c2a625f263855"}},{"code_sha256_prefix":"c1b9e144954a57e5","entry":"build_sam2_video_predictor","repo":"mark12ding/sam2long","repo_kind":"official","path":"sam2/build_sam.py","file_url":"https://github.com/mark12ding/sam2long/blob/HEAD/sam2/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1b9e144954a57e5"}},{"code_sha256_prefix":"d3df9ada3e05fa55","entry":"do_pool","repo":"mark12ding/sam2long","repo_kind":"official","path":"sam2/modeling/backbones/hieradet.py","file_url":"https://github.com/mark12ding/sam2long/blob/HEAD/sam2/modeling/backbones/hieradet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d3df9ada3e05fa55"}},{"code_sha256_prefix":"f2e72d81f39d65cd","entry":"get_activation_fn","repo":"mark12ding/sam2long","repo_kind":"official","path":"sam2/modeling/sam2_utils.py","file_url":"https://github.com/mark12ding/sam2long/blob/HEAD/sam2/modeling/sam2_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2e72d81f39d65cd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}