{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/segment-anything-meets-point-tracking","title":"Segment Anything Meets Point Tracking","arxiv_id":"2307.01197","date":"2023-07-03","proceeding":null,"authors":["Frano Rajič","Lei Ke","Yu-Wing Tai","Chi-Keung Tang","Martin Danelljan","Fisher Yu"],"abstract":"The Segment Anything Model (SAM) has established itself as a powerful zero-shot image segmentation model, enabled by efficient point-centric annotation and prompt-based models. While click and brush interactions are both well explored in interactive image segmentation, the existing methods on videos focus on mask annotation and propagation. This paper presents SAM-PT, a novel method for point-centric interactive video segmentation, empowered by SAM and long-term point tracking. SAM-PT leverages robust and sparse point selection and propagation techniques for mask generation. Compared to traditional object-centric mask propagation strategies, we uniquely use point propagation to exploit local structure information agnostic to object semantics. We highlight the merits of point-based tracking through direct evaluation on the zero-shot open-world Unidentified Video Objects (UVO) benchmark. Our experiments on popular video object segmentation and multi-object segmentation tracking benchmarks, including DAVIS, YouTube-VOS, and BDD100K, suggest that a point-based segmentation tracker yields better zero-shot performance and efficient interactions. We release our code that integrates different point trackers and video segmentation benchmarks at https://github.com/SysCV/sam-pt.","url_abs":"https://arxiv.org/abs/2307.01197v2","url_pdf":"https://arxiv.org/pdf/2307.01197v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"segment-anything-meets-point-tracking","repo_url":"https://github.com/syscv/sam-pt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"interactive-video-object-segmentation","task_name":"Interactive Video Object Segmentation"},{"task_slug":"object","task_name":"Object"},{"task_slug":"point-tracking","task_name":"Point Tracking"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"video-instance-segmentation","task_name":"Video Instance Segmentation"},{"task_slug":"video-object-segmentation","task_name":"Video Object Segmentation"},{"task_slug":"video-segmentation","task_name":"Video Segmentation"},{"task_slug":null,"task_name":"Zero-Shot Video Object Segmentation"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.01197","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.01197"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/syscv/sam-pt","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7e20bee8c9ab288b","entry":"add_mask_to_frame","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/utils/util.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e20bee8c9ab288b"}},{"code_sha256_prefix":"1776176a97dd455a","entry":"batchify_list","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/utils/util.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1776176a97dd455a"}},{"code_sha256_prefix":"fa8999c2e2fb539f","entry":"conv_channels_mixer","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/point_tracker/tapir/tapir_model.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/point_tracker/tapir/tapir_model.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa8999c2e2fb539f"}},{"code_sha256_prefix":"6d1df288856f878c","entry":"depthwise_conv_residual","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/point_tracker/tapir/tapir_model.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/point_tracker/tapir/tapir_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6d1df288856f878c"}},{"code_sha256_prefix":"082bcd7cae968600","entry":"extract_random_mask_points","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/utils/query_points.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/utils/query_points.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"082bcd7cae968600"}},{"code_sha256_prefix":"9d9402c814a980c3","entry":"get_linear_schedule_with_warmup","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/utils/util.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d9402c814a980c3"}},{"code_sha256_prefix":"3c5028c1ec6827e1","entry":"heatmaps_to_points","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/point_tracker/tapnet/tapnet_model.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/point_tracker/tapnet/tapnet_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c5028c1ec6827e1"}},{"code_sha256_prefix":"e5e6908904107ff9","entry":"interp","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/point_tracker/tapnet/tapnet_model.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/point_tracker/tapnet/tapnet_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e5e6908904107ff9"}},{"code_sha256_prefix":"64c885d6b9f4fd57","entry":"layernorm","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/point_tracker/tapir/tapir_model.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/point_tracker/tapir/tapir_model.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"64c885d6b9f4fd57"}},{"code_sha256_prefix":"2d2918c5490472b1","entry":"soft_argmax_heatmap","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/point_tracker/tapnet/tapnet_model.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/point_tracker/tapnet/tapnet_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d2918c5490472b1"}},{"code_sha256_prefix":"bd92506953233c24","entry":"visualize_int","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/modeling/sam_pt_interactive.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/modeling/sam_pt_interactive.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd92506953233c24"}},{"code_sha256_prefix":"c207e26ec20cc1e3","entry":"visualize_pred_against_gt_mask","repo":"syscv/sam-pt","repo_kind":"official","path":"sam_pt/modeling/sam_pt_interactive.py","file_url":"https://github.com/syscv/sam-pt/blob/HEAD/sam_pt/modeling/sam_pt_interactive.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c207e26ec20cc1e3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}