{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/univs-unified-and-universal-video","title":"UniVS: Unified and Universal Video Segmentation with Prompts as Queries","arxiv_id":"2402.18115","date":"2024-02-28","proceeding":"CVPR 2024 1","authors":["Minghan Li","Shuai Li","Xindong Zhang","Lei Zhang"],"abstract":"Despite the recent advances in unified image segmentation (IS), developing a unified video segmentation (VS) model remains a challenge. This is mainly because generic category-specified VS tasks need to detect all objects and track them across consecutive frames, while prompt-guided VS tasks require re-identifying the target with visual/text prompts throughout the entire video, making it hard to handle the different tasks with the same architecture. We make an attempt to address these issues and present a novel unified VS architecture, namely UniVS, by using prompts as queries. UniVS averages the prompt features of the target from previous frames as its initial query to explicitly decode masks, and introduces a target-wise prompt cross-attention layer in the mask decoder to integrate prompt features in the memory pool. By taking the predicted masks of entities from previous frames as their visual prompts, UniVS converts different VS tasks into prompt-guided target segmentation, eliminating the heuristic inter-frame matching process. Our framework not only unifies the different VS tasks but also naturally achieves universal training and testing, ensuring robust performance across different scenarios. UniVS shows a commendable balance between performance and universality on 10 challenging VS benchmarks, covering video instance, semantic, panoptic, object, and referring segmentation tasks. Code can be found at \\url{https://github.com/MinghanLi/UniVS}.","url_abs":"https://arxiv.org/abs/2402.18115v2","url_pdf":"https://arxiv.org/pdf/2402.18115v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"univs-unified-and-universal-video","repo_url":"https://github.com/minghanli/univs","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"referring-expression-segmentation","task_name":"Referring Expression Segmentation"},{"task_slug":"referring-video-object-segmentation","task_name":"Referring Video Object Segmentation"},{"task_slug":"video-instance-segmentation","task_name":"Video Instance Segmentation"},{"task_slug":"video-object-segmentation","task_name":"Video Object Segmentation"},{"task_slug":"video-object-tracking","task_name":"Video Object Tracking"},{"task_slug":"video-panoptic-segmentation","task_name":"Video Panoptic Segmentation"},{"task_slug":"video-segmentation","task_name":"Video Segmentation"},{"task_slug":"video-semantic-segmentation","task_name":"Video Semantic Segmentation"},{"task_slug":null,"task_name":"Zero-Shot Video Object Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/referring-expression-segmentation-on-davis","task":"Referring Expression Segmentation","dataset":"DAVIS 2017 (val)","model":"UniVS(Swin-L)","rank_in_archive_order":17,"of":18,"metrics":{"J&F 1st frame":"59.4?","J&F Full video":"59.4"},"uses_additional_data":true},{"leaderboard":"/sota/referring-expression-segmentation-on-refer-1","task":"Referring Expression Segmentation","dataset":"Refer-YouTube-VOS (2021 public validation)","model":"UniVS(Swin-L)","rank_in_archive_order":24,"of":33,"metrics":{"F":"59.5","J":"56.8","J&F":"58.0"},"uses_additional_data":true},{"leaderboard":"/sota/video-instance-segmentation-on-ovis-1","task":"Video Instance Segmentation","dataset":"OVIS validation","model":"UniVS(Swin-L)","rank_in_archive_order":18,"of":44,"metrics":{"mask AP":"41.7"},"uses_additional_data":true},{"leaderboard":"/sota/video-instance-segmentation-on-youtube-vis-2","task":"Video Instance Segmentation","dataset":"YouTube-VIS 2021","model":"UniVS(Swin-L)","rank_in_archive_order":12,"of":26,"metrics":{"AP50":"79.4","AP75":"63.3","AR1":"46.2","AR10":"63.1","mask AP":"57.9"},"uses_additional_data":true},{"leaderboard":"/sota/video-instance-segmentation-on-youtube-vis-1","task":"Video Instance Segmentation","dataset":"YouTube-VIS validation","model":"UniVS(Swin-L)","rank_in_archive_order":7,"of":44,"metrics":{"AP50":"82.1","AP75":"65.3","AR1":"54.7","AR10":"66.8","mask AP":"60.0"},"uses_additional_data":true},{"leaderboard":"/sota/video-object-segmentation-on-davis-2017-val","task":"Video Object Segmentation","dataset":"DAVIS 2017 (val)","model":"UniVS(Swin-L)","rank_in_archive_order":14,"of":17,"metrics":{"F-measure":"79.5","Jaccard":"72.8","Mean Jaccard & F-Measure":"76.2"},"uses_additional_data":true},{"leaderboard":"/sota/video-object-segmentation-on-youtube-vos-1","task":"Video Object Segmentation","dataset":"YouTube-VOS 2018","model":"UniVS(Swin-L)","rank_in_archive_order":13,"of":17,"metrics":{"Mean Jaccard & F-Measure":"71.5"},"uses_additional_data":true},{"leaderboard":"/sota/video-panoptic-segmentation-on-vipseg","task":"Video Panoptic Segmentation","dataset":"VIPSeg","model":"UniVS(Swin-L)","rank_in_archive_order":8,"of":12,"metrics":{"STQ":"58.2","VPQ":"49.3"},"uses_additional_data":true},{"leaderboard":"/sota/video-semantic-segmentation-on-vspw","task":"Video Semantic Segmentation","dataset":"VSPW","model":"UniVS(Swin-L)","rank_in_archive_order":2,"of":5,"metrics":{"mIoU":"59.8"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2402.18115","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.18115"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/minghanli/univs","reach":{"status":"ok"}}],"summary":{"ran":12,"unverified":2},"by_repo_kind":{"official":{"samples":14,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"b4a2ff2dcc61c4d4","entry":"batch_dice_coefficient_loss","repo":"minghanli/univs","repo_kind":"official","path":"univs/modeling/video_matcher.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/modeling/video_matcher.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4a2ff2dcc61c4d4"}},{"code_sha256_prefix":"a03c334f626ed752","entry":"batch_dice_loss","repo":"minghanli/univs","repo_kind":"official","path":"univs/modeling/video_matcher.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/modeling/video_matcher.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a03c334f626ed752"}},{"code_sha256_prefix":"0c3ac5c5ab657973","entry":"contrastive_aux_loss","repo":"minghanli/univs","repo_kind":"official","path":"univs/univs_prompt_longvideo.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/univs_prompt_longvideo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c3ac5c5ab657973"}},{"code_sha256_prefix":"bb2f0ddf9840604b","entry":"contrastive_loss","repo":"minghanli/univs","repo_kind":"official","path":"univs/univs_prompt_longvideo.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/univs_prompt_longvideo.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb2f0ddf9840604b"}},{"code_sha256_prefix":"e9937e82c9c52d3d","entry":"convert_box_to_mask","repo":"minghanli/univs","repo_kind":"official","path":"univs/univs_prompt.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/univs_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9937e82c9c52d3d"}},{"code_sha256_prefix":"63d3c6e9af5b08f8","entry":"dice_loss","repo":"minghanli/univs","repo_kind":"official","path":"univs/modeling/video_criterion.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/modeling/video_criterion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"63d3c6e9af5b08f8"}},{"code_sha256_prefix":"535792a3fc3afb99","entry":"dice_loss","repo":"minghanli/univs","repo_kind":"official","path":"univs/modeling/video_criterion_prompt.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/modeling/video_criterion_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"535792a3fc3afb99"}},{"code_sha256_prefix":"92c5b183ff2beae6","entry":"is_semseg_dataset","repo":"minghanli/univs","repo_kind":"official","path":"univs/prepare_targets.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/prepare_targets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92c5b183ff2beae6"}},{"code_sha256_prefix":"450b62615caa903a","entry":"sigmoid_ce_loss","repo":"minghanli/univs","repo_kind":"official","path":"univs/modeling/video_criterion.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/modeling/video_criterion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"450b62615caa903a"}},{"code_sha256_prefix":"46b849406325dd41","entry":"sigmoid_ce_loss","repo":"minghanli/univs","repo_kind":"official","path":"univs/modeling/video_criterion_prompt.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/modeling/video_criterion_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46b849406325dd41"}},{"code_sha256_prefix":"db2f336362cda7c8","entry":"sigmoid_ce_with_weight_loss","repo":"minghanli/univs","repo_kind":"official","path":"univs/modeling/video_criterion.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/modeling/video_criterion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db2f336362cda7c8"}},{"code_sha256_prefix":"1a3d5d8a73ce293d","entry":"sigmoid_ce_wo_logits_loss","repo":"minghanli/univs","repo_kind":"official","path":"univs/modeling/video_criterion_prompt.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/modeling/video_criterion_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a3d5d8a73ce293d"}},{"code_sha256_prefix":"1edd24985036b0bf","entry":"batch_sigmoid_ce_loss","repo":"minghanli/univs","repo_kind":"official","path":"univs/modeling/video_matcher.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/modeling/video_matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1edd24985036b0bf"}},{"code_sha256_prefix":"770dbf8ac853d862","entry":"copy_TeacherNet_weights","repo":"minghanli/univs","repo_kind":"official","path":"univs/utils/copy_TeacherNet_weights.py","file_url":"https://github.com/minghanli/univs/blob/HEAD/univs/utils/copy_TeacherNet_weights.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"770dbf8ac853d862"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}