{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/omnivid-a-generative-framework-for-universal","title":"OmniVid: A Generative Framework for Universal Video Understanding","arxiv_id":"2403.17935","date":"2024-03-26","proceeding":"CVPR 2024 1","authors":["Junke Wang","Dongdong Chen","Chong Luo","Bo He","Lu Yuan","Zuxuan Wu","Yu-Gang Jiang"],"abstract":"The core of video understanding tasks, such as recognition, captioning, and tracking, is to automatically detect objects or actions in a video and analyze their temporal evolution. Despite sharing a common goal, different tasks often rely on distinct model architectures and annotation formats. In contrast, natural language processing benefits from a unified output space, i.e., text sequences, which simplifies the training of powerful foundational language models, such as GPT-3, with extensive training corpora. Inspired by this, we seek to unify the output space of video understanding tasks by using languages as labels and additionally introducing time and box tokens. In this way, a variety of video tasks could be formulated as video-grounded token generation. This enables us to address various types of video tasks, including classification (such as action recognition), captioning (covering clip captioning, video question answering, and dense video captioning), and localization tasks (such as visual object tracking) within a fully shared encoder-decoder architecture, following a generative framework. Through comprehensive experiments, we demonstrate such a simple and straightforward idea is quite effective and can achieve state-of-the-art or competitive results on seven video benchmarks, providing a novel perspective for more universal video understanding. Code is available at https://github.com/wangjk666/OmniVid.","url_abs":"https://arxiv.org/abs/2403.17935v1","url_pdf":"https://arxiv.org/pdf/2403.17935v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"omnivid-a-generative-framework-for-universal","repo_url":"https://github.com/wangjk666/omnivid","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"dense-video-captioning","task_name":"Dense Video Captioning"},{"task_slug":"object-tracking","task_name":"Object Tracking"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-understanding","task_name":"Video Understanding"},{"task_slug":"visual-object-tracking","task_name":"Visual Object Tracking"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.17935","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.17935"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/wangjk666/OmniVid","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":8},"by_repo_kind":{"official":{"samples":11,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"b446359db2c939d6","entry":"SIoU_loss","repo":"wangjk666/OmniVid","repo_kind":"official","path":"od_util/box_loss.py","file_url":"https://github.com/wangjk666/OmniVid/blob/HEAD/od_util/box_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b446359db2c939d6"}},{"code_sha256_prefix":"5f8ef4b125efb49f","entry":"fp16_clamp","repo":"wangjk666/OmniVid","repo_kind":"official","path":"od_util/box_loss.py","file_url":"https://github.com/wangjk666/OmniVid/blob/HEAD/od_util/box_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f8ef4b125efb49f"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"wangjk666/OmniVid","repo_kind":"official","path":"lavis/models/base_model.py","file_url":"https://github.com/wangjk666/OmniVid/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"wangjk666/OmniVid","repo_kind":"official","path":"lavis/models/base_model.py","file_url":"https://github.com/wangjk666/OmniVid/blob/HEAD/lavis/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"4373b99bf6b5fd7d","entry":"ciou","repo":"wangjk666/OmniVid","repo_kind":"official","path":"od_util/box_loss.py","file_url":"https://github.com/wangjk666/OmniVid/blob/HEAD/od_util/box_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4373b99bf6b5fd7d"}},{"code_sha256_prefix":"4f25251c868cedb9","entry":"eval_soda","repo":"wangjk666/OmniVid","repo_kind":"official","path":"densevid_eval3/eval_soda.py","file_url":"https://github.com/wangjk666/OmniVid/blob/HEAD/densevid_eval3/eval_soda.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4f25251c868cedb9"}},{"code_sha256_prefix":"02cf5c2eb41e68fa","entry":"eval_tool","repo":"wangjk666/OmniVid","repo_kind":"official","path":"densevid_eval3/eval_soda.py","file_url":"https://github.com/wangjk666/OmniVid/blob/HEAD/densevid_eval3/eval_soda.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"02cf5c2eb41e68fa"}},{"code_sha256_prefix":"4b6ebfb6c53af660","entry":"parse_para","repo":"wangjk666/OmniVid","repo_kind":"official","path":"densevid_eval3/para_evaluate.py","file_url":"https://github.com/wangjk666/OmniVid/blob/HEAD/densevid_eval3/para_evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b6ebfb6c53af660"}},{"code_sha256_prefix":"36a9ad5fe7008b51","entry":"parse_sent","repo":"wangjk666/OmniVid","repo_kind":"official","path":"densevid_eval3/para_evaluate.py","file_url":"https://github.com/wangjk666/OmniVid/blob/HEAD/densevid_eval3/para_evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36a9ad5fe7008b51"}},{"code_sha256_prefix":"cfb0aa2342a41918","entry":"random_string","repo":"wangjk666/OmniVid","repo_kind":"official","path":"densevid_eval3/evaluate2021.py","file_url":"https://github.com/wangjk666/OmniVid/blob/HEAD/densevid_eval3/evaluate2021.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfb0aa2342a41918"}},{"code_sha256_prefix":"a45f62024eb3d2c5","entry":"remove_nonascii","repo":"wangjk666/OmniVid","repo_kind":"official","path":"densevid_eval3/evaluate2018.py","file_url":"https://github.com/wangjk666/OmniVid/blob/HEAD/densevid_eval3/evaluate2018.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a45f62024eb3d2c5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}