{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/video-mask-transfiner-for-high-quality-video","title":"Video Mask Transfiner for High-Quality Video Instance Segmentation","arxiv_id":"2207.14012","date":"2022-07-28","proceeding":null,"authors":["Lei Ke","Henghui Ding","Martin Danelljan","Yu-Wing Tai","Chi-Keung Tang","Fisher Yu"],"abstract":"While Video Instance Segmentation (VIS) has seen rapid progress, current approaches struggle to predict high-quality masks with accurate boundary details. Moreover, the predicted segmentations often fluctuate over time, suggesting that temporal consistency cues are neglected or not fully utilized. In this paper, we set out to tackle these issues, with the aim of achieving highly detailed and more temporally stable mask predictions for VIS. We first propose the Video Mask Transfiner (VMT) method, capable of leveraging fine-grained high-resolution features thanks to a highly efficient video transformer structure. Our VMT detects and groups sparse error-prone spatio-temporal regions of each tracklet in the video segment, which are then refined using both local and instance-level cues. Second, we identify that the coarse boundary annotations of the popular YouTube-VIS dataset constitute a major limiting factor. Based on our VMT architecture, we therefore design an automated annotation refinement approach by iterative training and self-correction. To benchmark high-quality mask predictions for VIS, we introduce the HQ-YTVIS dataset, consisting of a manually re-annotated test set and our automatically refined training data. We compare VMT with the most recent state-of-the-art methods on the HQ-YTVIS, as well as the Youtube-VIS, OVIS and BDD100K MOTS benchmarks. Experimental results clearly demonstrate the efficacy and effectiveness of our method on segmenting complex and dynamic objects, by capturing precise details.","url_abs":"https://arxiv.org/abs/2207.14012v1","url_pdf":"https://arxiv.org/pdf/2207.14012v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"video-mask-transfiner-for-high-quality-video","repo_url":"https://github.com/SysCV/vmt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"video-instance-segmentation","task_name":"Video Instance Segmentation"},{"task_slug":"high","task_name":"Vocal Bursts Intensity Prediction"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[{"slug":"hq-ytvis","name":"HQ-YTVIS","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-instance-segmentation-on-hq-ytvis","task":"Video Instance Segmentation","dataset":"HQ-YTVIS","model":"VMT (Swin-L)","rank_in_archive_order":1,"of":4,"metrics":{"Tube-Boundary AP":"44.8"},"uses_additional_data":false},{"leaderboard":"/sota/video-instance-segmentation-on-hq-ytvis","task":"Video Instance Segmentation","dataset":"HQ-YTVIS","model":"VMT (R101)","rank_in_archive_order":3,"of":4,"metrics":{"Tube-Boundary AP":"32.5"},"uses_additional_data":false},{"leaderboard":"/sota/video-instance-segmentation-on-hq-ytvis","task":"Video Instance Segmentation","dataset":"HQ-YTVIS","model":"VMT (R50)","rank_in_archive_order":4,"of":4,"metrics":{"Tube-Boundary AP":"30.7"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2207.14012","atlas_url":"https://app.syntology.ai/?focus=2207.14012","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.14012"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SysCV/vmt","reach":null}],"summary":{"ran":2,"ran_draft_wrong":3,"unverified":2},"by_repo_kind":{"listed":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"501c57cf59365f88","entry":"TransformerEncoder","repo":"SysCV/vmt","repo_kind":"listed","path":"models/segmentation.py","file_url":"https://github.com/SysCV/vmt/blob/HEAD/models/segmentation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"501c57cf59365f88"}},{"code_sha256_prefix":"ed01e90c38f337c9","entry":"TransformerEncoderLayer","repo":"SysCV/vmt","repo_kind":"listed","path":"models/segmentation.py","file_url":"https://github.com/SysCV/vmt/blob/HEAD/models/segmentation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed01e90c38f337c9"}},{"code_sha256_prefix":"27463d5407d3dddf","entry":"nested_tensor_from_tensor_list","repo":"SysCV/vmt","repo_kind":"listed","path":"models/segmentation.py","file_url":"https://github.com/SysCV/vmt/blob/HEAD/models/segmentation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"27463d5407d3dddf"}},{"code_sha256_prefix":"46f34dc9f9a94b67","entry":"parse_dynamic_params","repo":"SysCV/vmt","repo_kind":"listed","path":"models/segmentation.py","file_url":"https://github.com/SysCV/vmt/blob/HEAD/models/segmentation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"46f34dc9f9a94b67"}},{"code_sha256_prefix":"ca652a3292b6646f","entry":"pos_embed","repo":"SysCV/vmt","repo_kind":"listed","path":"models/segmentation.py","file_url":"https://github.com/SysCV/vmt/blob/HEAD/models/segmentation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca652a3292b6646f"}},{"code_sha256_prefix":"e3aeaa4531ba2770","entry":"MaskHeadSmallConv","repo":"SysCV/vmt","repo_kind":"listed","path":"models/segmentation.py","file_url":"https://github.com/SysCV/vmt/blob/HEAD/models/segmentation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e3aeaa4531ba2770"}},{"code_sha256_prefix":"aa70e8fcfd59bba9","entry":"VMT","repo":"SysCV/vmt","repo_kind":"listed","path":"models/segmentation.py","file_url":"https://github.com/SysCV/vmt/blob/HEAD/models/segmentation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa70e8fcfd59bba9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}