{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/univtg-towards-unified-video-language","title":"UniVTG: Towards Unified Video-Language Temporal Grounding","arxiv_id":"2307.16715","date":"2023-07-31","proceeding":"ICCV 2023 1","authors":["Kevin Qinghong Lin","Pengchuan Zhang","Joya Chen","Shraman Pramanick","Difei Gao","Alex Jinpeng Wang","Rui Yan","Mike Zheng Shou"],"abstract":"Video Temporal Grounding (VTG), which aims to ground target clips from videos (such as consecutive intervals or disjoint shots) according to custom language queries (e.g., sentences or words), is key for video browsing on social media. Most methods in this direction develop taskspecific models that are trained with type-specific labels, such as moment retrieval (time interval) and highlight detection (worthiness curve), which limits their abilities to generalize to various VTG tasks and labels. In this paper, we propose to Unify the diverse VTG labels and tasks, dubbed UniVTG, along three directions: Firstly, we revisit a wide range of VTG labels and tasks and define a unified formulation. Based on this, we develop data annotation schemes to create scalable pseudo supervision. Secondly, we develop an effective and flexible grounding model capable of addressing each task and making full use of each label. Lastly, thanks to the unified framework, we are able to unlock temporal grounding pretraining from large-scale diverse labels and develop stronger grounding abilities e.g., zero-shot grounding. Extensive experiments on three tasks (moment retrieval, highlight detection and video summarization) across seven datasets (QVHighlights, Charades-STA, TACoS, Ego4D, YouTube Highlights, TVSum, and QFVS) demonstrate the effectiveness and flexibility of our proposed framework. The codes are available at https://github.com/showlab/UniVTG.","url_abs":"https://arxiv.org/abs/2307.16715v2","url_pdf":"https://arxiv.org/pdf/2307.16715v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"univtg-towards-unified-video-language","repo_url":"https://github.com/showlab/univtg","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"highlight-detection","task_name":"Highlight Detection"},{"task_slug":"moment-retrieval","task_name":"Moment Retrieval"},{"task_slug":"natural-language-moment-retrieval","task_name":"Natural Language Moment Retrieval"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-summarization","task_name":"Video Summarization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/highlight-detection-on-qvhighlights","task":"Highlight Detection","dataset":"QVHighlights","model":"UniVTG (w/ PT)","rank_in_archive_order":9,"of":21,"metrics":{"Hit@1":"66.28","mAP":"40.54"},"uses_additional_data":true},{"leaderboard":"/sota/highlight-detection-on-qvhighlights","task":"Highlight Detection","dataset":"QVHighlights","model":"UniVTG","rank_in_archive_order":16,"of":21,"metrics":{"Hit@1":"60.96","mAP":"38.20"},"uses_additional_data":false},{"leaderboard":"/sota/moment-retrieval-on-qvhighlights","task":"Moment Retrieval","dataset":"QVHighlights","model":"UniVTG (w/ PT)","rank_in_archive_order":17,"of":32,"metrics":{"R@1 IoU=0.5":"65.43","R@1 IoU=0.7":"50.06","mAP":"43.63","mAP@0.5":"64.06","mAP@0.75":"45.02"},"uses_additional_data":true},{"leaderboard":"/sota/moment-retrieval-on-qvhighlights","task":"Moment Retrieval","dataset":"QVHighlights","model":"UniVTG","rank_in_archive_order":28,"of":32,"metrics":{"R@1 IoU=0.5":"58.86","R@1 IoU=0.7":"40.86","mAP":"35.47","mAP@0.5":"57.60","mAP@0.75":"35.59"},"uses_additional_data":false},{"leaderboard":"/sota/natural-language-moment-retrieval-on-tacos","task":"Natural Language Moment Retrieval","dataset":"TACoS","model":"UniVTG","rank_in_archive_order":9,"of":13,"metrics":{"R@1,IoU=0.3":"51.44","R@1,IoU=0.5":"34.97","R@1,IoU=0.7":"21.07","mIoU":"35.76"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2307.16715","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.16715"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/showlab/univtg","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/showlab/UniVTG","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"ran_fixture":1,"ran_honours":1,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":16,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fe525fce2ed45fdc","entry":"Conv","repo":"showlab/UniVTG","repo_kind":"official","path":"model/univtg.py","file_url":"https://github.com/showlab/UniVTG/blob/HEAD/model/univtg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe525fce2ed45fdc"}},{"code_sha256_prefix":"71a63e8e4005103c","entry":"Model","repo":"showlab/UniVTG","repo_kind":"official","path":"model/univtg.py","file_url":"https://github.com/showlab/UniVTG/blob/HEAD/model/univtg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71a63e8e4005103c"}},{"code_sha256_prefix":"94aab69ebd3439ec","entry":"PositionalEncoding","repo":"showlab/univtg","repo_kind":"official","path":"model/position_encoding.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/model/position_encoding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94aab69ebd3439ec"}},{"code_sha256_prefix":"2912d21d62514265","entry":"build_position_encoding","repo":"showlab/univtg","repo_kind":"official","path":"model/position_encoding.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/model/position_encoding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2912d21d62514265"}},{"code_sha256_prefix":"0dc91e6607bb2ec1","entry":"convert_to_hms","repo":"showlab/univtg","repo_kind":"official","path":"main_gradio.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/main_gradio.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0dc91e6607bb2ec1"}},{"code_sha256_prefix":"10e67144852e5fce","entry":"drop_path","repo":"showlab/univtg","repo_kind":"official","path":"model/transformer_encoder_droppath.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/model/transformer_encoder_droppath.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10e67144852e5fce"}},{"code_sha256_prefix":"60df4d5a1eee1b92","entry":"mask_logits","repo":"showlab/univtg","repo_kind":"official","path":"model/univtg.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/model/univtg.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60df4d5a1eee1b92"}},{"code_sha256_prefix":"4d2286148f711e02","entry":"process_mat","repo":"showlab/univtg","repo_kind":"official","path":"eval/qfvs.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/eval/qfvs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d2286148f711e02"}},{"code_sha256_prefix":"b3a40dcb09d6a4a9","entry":"process_video_mat","repo":"showlab/univtg","repo_kind":"official","path":"eval/qfvs.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/eval/qfvs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3a40dcb09d6a4a9"}},{"code_sha256_prefix":"185c7b880ca6135e","entry":"sim_matrix","repo":"showlab/univtg","repo_kind":"official","path":"model/univtg.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/model/univtg.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"185c7b880ca6135e"}},{"code_sha256_prefix":"081c8abd57d082fd","entry":"accuracy","repo":"showlab/univtg","repo_kind":"official","path":"model/moment_detr.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/model/moment_detr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"081c8abd57d082fd"}},{"code_sha256_prefix":"a03d2e1a023a5c63","entry":"build_transformer","repo":"showlab/univtg","repo_kind":"official","path":"model/transformer.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/model/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a03d2e1a023a5c63"}},{"code_sha256_prefix":"141ae5fb5b4041f0","entry":"build_transformer","repo":"showlab/univtg","repo_kind":"official","path":"model/transformer_encoder_droppath.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/model/transformer_encoder_droppath.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"141ae5fb5b4041f0"}},{"code_sha256_prefix":"edb929dece68582e","entry":"count_parameters","repo":"showlab/univtg","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edb929dece68582e"}},{"code_sha256_prefix":"6846bf0b5c7629d7","entry":"init_weights","repo":"showlab/UniVTG","repo_kind":"official","path":"model/univtg.py","file_url":"https://github.com/showlab/UniVTG/blob/HEAD/model/univtg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6846bf0b5c7629d7"}},{"code_sha256_prefix":"0581196da1c218b5","entry":"load_videos_tag","repo":"showlab/univtg","repo_kind":"official","path":"eval/qfvs.py","file_url":"https://github.com/showlab/univtg/blob/HEAD/eval/qfvs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0581196da1c218b5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}