{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/devis-making-deformable-transformers-work-for","title":"DeVIS: Making Deformable Transformers Work for Video Instance Segmentation","arxiv_id":"2207.11103","date":"2022-07-22","proceeding":null,"authors":["Adrià Caelles","Tim Meinhardt","Guillem Brasó","Laura Leal-Taixé"],"abstract":"Video Instance Segmentation (VIS) jointly tackles multi-object detection, tracking, and segmentation in video sequences. In the past, VIS methods mirrored the fragmentation of these subtasks in their architectural design, hence missing out on a joint solution. Transformers recently allowed to cast the entire VIS task as a single set-prediction problem. Nevertheless, the quadratic complexity of existing Transformer-based methods requires long training times, high memory requirements, and processing of low-single-scale feature maps. Deformable attention provides a more efficient alternative but its application to the temporal domain or the segmentation task have not yet been explored. In this work, we present Deformable VIS (DeVIS), a VIS method which capitalizes on the efficiency and performance of deformable Transformers. To reason about all VIS subtasks jointly over multiple frames, we present temporal multi-scale deformable attention with instance-aware object queries. We further introduce a new image and video instance mask head with multi-scale features, and perform near-online video processing with multi-cue clip tracking. DeVIS reduces memory as well as training time requirements, and achieves state-of-the-art results on the YouTube-VIS 2021, as well as the challenging OVIS dataset. Code is available at https://github.com/acaelles97/DeVIS.","url_abs":"https://arxiv.org/abs/2207.11103v1","url_pdf":"https://arxiv.org/pdf/2207.11103v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"devis-making-deformable-transformers-work-for","repo_url":"https://github.com/acaelles97/devis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"video-instance-segmentation","task_name":"Video Instance Segmentation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-instance-segmentation-on-ovis-1","task":"Video Instance Segmentation","dataset":"OVIS validation","model":"DeVIS (Swin-L)","rank_in_archive_order":25,"of":44,"metrics":{"AP50":"59.3","AP75":"38.3","AR1":"16.6","AR10":"39.8","mask AP":"35.5"},"uses_additional_data":false},{"leaderboard":"/sota/video-instance-segmentation-on-ovis-1","task":"Video Instance Segmentation","dataset":"OVIS validation","model":"DeVIS (ResNet-50)","rank_in_archive_order":33,"of":44,"metrics":{"AP50":"47.6","AP75":"20.8","AR1":"12.0","AR10":"28.9","mask AP":"23.7"},"uses_additional_data":false},{"leaderboard":"/sota/video-instance-segmentation-on-youtube-vis-2","task":"Video Instance Segmentation","dataset":"YouTube-VIS 2021","model":"DeVIS (Swin-L)","rank_in_archive_order":17,"of":26,"metrics":{"AP50":"77.7","AP75":"59.8","AR1":"43.8","AR10":"57.8","mask AP":"54.4"},"uses_additional_data":false},{"leaderboard":"/sota/video-instance-segmentation-on-youtube-vis-2","task":"Video Instance Segmentation","dataset":"YouTube-VIS 2021","model":"DeVIS (ResNet-50)","rank_in_archive_order":24,"of":26,"metrics":{"AP50":"66.8","AP75":"46.6","AR1":"38.0","AR10":"50.1","mask AP":"43.1"},"uses_additional_data":false},{"leaderboard":"/sota/video-instance-segmentation-on-youtube-vis-1","task":"Video Instance Segmentation","dataset":"YouTube-VIS validation","model":"DeVIS (Swin-L)","rank_in_archive_order":10,"of":44,"metrics":{"AP50":"80.8","AP75":"66.3","AR1":"50.8","AR10":"61.0","mask AP":"57.1"},"uses_additional_data":false},{"leaderboard":"/sota/video-instance-segmentation-on-youtube-vis-1","task":"Video Instance Segmentation","dataset":"YouTube-VIS validation","model":"DeVIS (ResNet-50)","rank_in_archive_order":23,"of":44,"metrics":{"AP50":"66.7","AP75":"48.6","AR1":"42.4","AR10":"51.6","mask AP":"44.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.11103","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.11103"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/acaelles97/devis","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":9},"by_repo_kind":{"official":{"samples":11,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"f31a58bf6457ced5","entry":"convert_to_xywh","repo":"acaelles97/devis","repo_kind":"official","path":"src/datasets/coco_eval.py","file_url":"https://github.com/acaelles97/devis/blob/HEAD/src/datasets/coco_eval.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f31a58bf6457ced5"}},{"code_sha256_prefix":"fe0ddcc2d420c9a0","entry":"evaluate","repo":"acaelles97/devis","repo_kind":"official","path":"src/datasets/coco_eval.py","file_url":"https://github.com/acaelles97/devis/blob/HEAD/src/datasets/coco_eval.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"fe0ddcc2d420c9a0"}},{"code_sha256_prefix":"894e7d5a12a809b7","entry":"build_3D_conv_head","repo":"acaelles97/devis","repo_kind":"official","path":"src/models/devis_segmentation.py","file_url":"https://github.com/acaelles97/devis/blob/HEAD/src/models/devis_segmentation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"894e7d5a12a809b7"}},{"code_sha256_prefix":"8ea3bd3cae92680d","entry":"crop","repo":"acaelles97/devis","repo_kind":"official","path":"src/datasets/coco_transforms.py","file_url":"https://github.com/acaelles97/devis/blob/HEAD/src/datasets/coco_transforms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8ea3bd3cae92680d"}},{"code_sha256_prefix":"2354640608431f61","entry":"get_size_with_aspect_ratio","repo":"acaelles97/devis","repo_kind":"official","path":"src/datasets/coco_transforms.py","file_url":"https://github.com/acaelles97/devis/blob/HEAD/src/datasets/coco_transforms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2354640608431f61"}},{"code_sha256_prefix":"344591b983c871de","entry":"hflip","repo":"acaelles97/devis","repo_kind":"official","path":"src/datasets/coco_transforms.py","file_url":"https://github.com/acaelles97/devis/blob/HEAD/src/datasets/coco_transforms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"344591b983c871de"}},{"code_sha256_prefix":"9fc36974a286ba3c","entry":"make_coco_train_transforms","repo":"acaelles97/devis","repo_kind":"official","path":"src/datasets/coco.py","file_url":"https://github.com/acaelles97/devis/blob/HEAD/src/datasets/coco.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9fc36974a286ba3c"}},{"code_sha256_prefix":"08f78c9f2a7c5d05","entry":"make_coco_val_transforms","repo":"acaelles97/devis","repo_kind":"official","path":"src/datasets/coco.py","file_url":"https://github.com/acaelles97/devis/blob/HEAD/src/datasets/coco.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"08f78c9f2a7c5d05"}},{"code_sha256_prefix":"eb4edcefd813ed1f","entry":"merge","repo":"acaelles97/devis","repo_kind":"official","path":"src/datasets/coco_eval.py","file_url":"https://github.com/acaelles97/devis/blob/HEAD/src/datasets/coco_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"eb4edcefd813ed1f"}},{"code_sha256_prefix":"c1af59f873dcaf84","entry":"process_centroid","repo":"acaelles97/devis","repo_kind":"official","path":"visualize_dataset.py","file_url":"https://github.com/acaelles97/devis/blob/HEAD/visualize_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c1af59f873dcaf84"}},{"code_sha256_prefix":"96420c6020086b5c","entry":"un_normalize","repo":"acaelles97/devis","repo_kind":"official","path":"visualize_dataset.py","file_url":"https://github.com/acaelles97/devis/blob/HEAD/visualize_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"96420c6020086b5c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}