{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mask-free-video-instance-segmentation","title":"Mask-Free Video Instance Segmentation","arxiv_id":"2303.15904","date":"2023-03-28","proceeding":"CVPR 2023 1","authors":["Lei Ke","Martin Danelljan","Henghui Ding","Yu-Wing Tai","Chi-Keung Tang","Fisher Yu"],"abstract":"The recent advancement in Video Instance Segmentation (VIS) has largely been driven by the use of deeper and increasingly data-hungry transformer-based models. However, video masks are tedious and expensive to annotate, limiting the scale and diversity of existing VIS datasets. In this work, we aim to remove the mask-annotation requirement. We propose MaskFreeVIS, achieving highly competitive VIS performance, while only using bounding box annotations for the object state. We leverage the rich temporal mask consistency constraints in videos by introducing the Temporal KNN-patch Loss (TK-Loss), providing strong mask supervision without any labels. Our TK-Loss finds one-to-many matches across frames, through an efficient patch-matching step followed by a K-nearest neighbor selection. A consistency loss is then enforced on the found matches. Our mask-free objective is simple to implement, has no trainable parameters, is computationally efficient, yet outperforms baselines employing, e.g., state-of-the-art optical flow to enforce temporal mask consistency. We validate MaskFreeVIS on the YouTube-VIS 2019/2021, OVIS and BDD100K MOTS benchmarks. The results clearly demonstrate the efficacy of our method by drastically narrowing the gap between fully and weakly-supervised VIS performance. Our code and trained models are available at https://github.com/SysCV/MaskFreeVis.","url_abs":"https://arxiv.org/abs/2303.15904v1","url_pdf":"https://arxiv.org/pdf/2303.15904v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mask-free-video-instance-segmentation","repo_url":"https://github.com/syscv/maskfreevis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"patch-matching","task_name":"Patch Matching"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"video-instance-segmentation","task_name":"Video Instance Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-instance-segmentation-on-youtube-vis-4","task":"Video Instance Segmentation","dataset":"Youtube-VIS (trained with no video masks)","model":"MaskFreeVIS","rank_in_archive_order":1,"of":1,"metrics":{"AP":"55.3"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.15904","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.15904"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/syscv/maskfreevis","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"28180c8c241c50da","entry":"batch_dice_loss","repo":"syscv/maskfreevis","repo_kind":"official","path":"mask2former/modeling/matcher.py","file_url":"https://github.com/syscv/maskfreevis/blob/HEAD/mask2former/modeling/matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"28180c8c241c50da"}},{"code_sha256_prefix":"0528a6d568ba0384","entry":"batch_sigmoid_ce_loss","repo":"syscv/maskfreevis","repo_kind":"official","path":"mask2former/modeling/matcher.py","file_url":"https://github.com/syscv/maskfreevis/blob/HEAD/mask2former/modeling/matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0528a6d568ba0384"}},{"code_sha256_prefix":"b3fe5761e27f8013","entry":"compute_pairwise_term","repo":"syscv/maskfreevis","repo_kind":"official","path":"mask2former/modeling/criterion.py","file_url":"https://github.com/syscv/maskfreevis/blob/HEAD/mask2former/modeling/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b3fe5761e27f8013"}},{"code_sha256_prefix":"fc24cdf5cc5ceaa5","entry":"get_images_color_similarity","repo":"syscv/maskfreevis","repo_kind":"official","path":"mask2former/maskformer_model.py","file_url":"https://github.com/syscv/maskfreevis/blob/HEAD/mask2former/maskformer_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc24cdf5cc5ceaa5"}},{"code_sha256_prefix":"79ed1fef2ef4ba1b","entry":"get_incoherent_mask","repo":"syscv/maskfreevis","repo_kind":"official","path":"mask2former/modeling/criterion.py","file_url":"https://github.com/syscv/maskfreevis/blob/HEAD/mask2former/modeling/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"79ed1fef2ef4ba1b"}},{"code_sha256_prefix":"00a97608e14a1d96","entry":"masks_to_boxes","repo":"syscv/maskfreevis","repo_kind":"official","path":"mask2former/modeling/matcher.py","file_url":"https://github.com/syscv/maskfreevis/blob/HEAD/mask2former/modeling/matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"00a97608e14a1d96"}},{"code_sha256_prefix":"ae42eb0f61943a47","entry":"unfold_w_center","repo":"syscv/maskfreevis","repo_kind":"official","path":"mask2former_video/video_maskformer_model.py","file_url":"https://github.com/syscv/maskfreevis/blob/HEAD/mask2former_video/video_maskformer_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ae42eb0f61943a47"}},{"code_sha256_prefix":"461f78b78e4ae354","entry":"unfold_wo_center","repo":"syscv/maskfreevis","repo_kind":"official","path":"mask2former/maskformer_model.py","file_url":"https://github.com/syscv/maskfreevis/blob/HEAD/mask2former/maskformer_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"461f78b78e4ae354"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}