{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/video-instance-segmentation-using-inter-frame","title":"Video Instance Segmentation using Inter-Frame Communication Transformers","arxiv_id":"2106.03299","date":"2021-06-07","proceeding":"NeurIPS 2021 12","authors":["Sukjun Hwang","Miran Heo","Seoung Wug Oh","Seon Joo Kim"],"abstract":"We propose a novel end-to-end solution for video instance segmentation (VIS) based on transformers. Recently, the per-clip pipeline shows superior performance over per-frame methods leveraging richer information from multiple frames. However, previous per-clip models require heavy computation and memory usage to achieve frame-to-frame communications, limiting practicality. In this work, we propose Inter-frame Communication Transformers (IFC), which significantly reduces the overhead for information-passing between frames by efficiently encoding the context within the input clip. Specifically, we propose to utilize concise memory tokens as a mean of conveying information as well as summarizing each frame scene. The features of each frame are enriched and correlated with other frames through exchange of information between the precisely encoded memory tokens. We validate our method on the latest benchmark sets and achieved the state-of-the-art performance (AP 44.6 on YouTube-VIS 2019 val set using the offline inference) while having a considerably fast runtime (89.4 FPS). Our method can also be applied to near-online inference for processing a video in real-time with only a small delay. The code will be made available.","url_abs":"https://arxiv.org/abs/2106.03299v1","url_pdf":"https://arxiv.org/pdf/2106.03299v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"video-instance-segmentation-using-inter-frame","repo_url":"https://github.com/sukjunhwang/IFC","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"video-instance-segmentation","task_name":"Video Instance Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-instance-segmentation-on-youtube-vis-1","task":"Video Instance Segmentation","dataset":"YouTube-VIS validation","model":"IFC (ResNet-50)","rank_in_archive_order":24,"of":44,"metrics":{"AP50":"65.8","AP75":"46.8","AR1":"43.8","AR10":"51.2","mask AP":"42.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.03299","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.03299"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sukjunhwang/IFC","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_violates":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6296ea80154ae7f9","entry":"dice_coef","repo":"sukjunhwang/IFC","repo_kind":"official","path":"projects/IFC/ifc/models/segmentation.py","file_url":"https://github.com/sukjunhwang/IFC/blob/HEAD/projects/IFC/ifc/models/segmentation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6296ea80154ae7f9"}},{"code_sha256_prefix":"ac8fe530cdad4d8c","entry":"dice_loss","repo":"sukjunhwang/IFC","repo_kind":"official","path":"projects/IFC/ifc/models/segmentation.py","file_url":"https://github.com/sukjunhwang/IFC/blob/HEAD/projects/IFC/ifc/models/segmentation.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac8fe530cdad4d8c"}},{"code_sha256_prefix":"9502374fe12f11e4","entry":"sigmoid_focal_coef","repo":"sukjunhwang/IFC","repo_kind":"official","path":"projects/IFC/ifc/models/segmentation.py","file_url":"https://github.com/sukjunhwang/IFC/blob/HEAD/projects/IFC/ifc/models/segmentation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9502374fe12f11e4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}