{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/omni-video-democratizing-unified-video","title":"Omni-Video: Democratizing Unified Video Understanding and Generation","arxiv_id":"2507.06119","date":"2025-07-08","proceeding":null,"authors":["Zhiyu Tan","Hao Yang","Luozheng Qin","Jia Gong","Mengping Yang","Hao Li"],"abstract":"Notable breakthroughs in unified understanding and generation modeling have led to remarkable advancements in image understanding, reasoning, production and editing, yet current foundational models predominantly focus on processing images, creating a gap in the development of unified models for video understanding and generation. This report presents Omni-Video, an efficient and effective unified framework for video understanding, generation, as well as instruction-based editing. Our key insight is to teach existing multimodal large language models (MLLMs) to produce continuous visual clues that are used as the input of diffusion decoders, which produce high-quality videos conditioned on these visual clues. To fully unlock the potential of our system for unified video modeling, we integrate several technical improvements: 1) a lightweight architectural design that respectively attaches a vision head on the top of MLLMs and a adapter before the input of diffusion decoders, the former produce visual tokens for the latter, which adapts these visual tokens to the conditional space of diffusion decoders; and 2) an efficient multi-stage training scheme that facilitates a fast connection between MLLMs and diffusion decoders with limited data and computational resources. We empirically demonstrate that our model exhibits satisfactory generalization abilities across video generation, editing and understanding tasks.","url_abs":"https://arxiv.org/abs/2507.06119v2","url_pdf":"https://arxiv.org/pdf/2507.06119v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"omni-video-democratizing-unified-video","repo_url":"https://github.com/sais-fuxi/omni-video","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"video-generation","task_name":"Video Generation"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2507.06119","atlas_url":"https://app.syntology.ai/?focus=2507.06119","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2507.06119"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sais-fuxi/omni-video","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran_fixture":1,"unverified":15},"by_repo_kind":{"official":{"samples":18,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/tokenizers.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/tokenizers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"e34a97853e924d88","entry":"sinusoidal_embedding_1d","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/model.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/model.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e34a97853e924d88"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/tokenizers.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/tokenizers.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"e46ebd89452d4376","entry":"attention","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/attention.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e46ebd89452d4376"}},{"code_sha256_prefix":"1066f2140d4e9845","entry":"canonicalize","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/tokenizers.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/tokenizers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1066f2140d4e9845"}},{"code_sha256_prefix":"78d2ce0959095290","entry":"count_conv3d","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/vae2_1.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/vae2_1.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"78d2ce0959095290"}},{"code_sha256_prefix":"c2098ac910a95bc5","entry":"flash_attention","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/attention.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c2098ac910a95bc5"}},{"code_sha256_prefix":"d3d53f877a9af3a3","entry":"fp16_clamp","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/t5.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/t5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3d53f877a9af3a3"}},{"code_sha256_prefix":"169bb5776300d2eb","entry":"generate_source_video_caption","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/vllm_model.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/vllm_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"169bb5776300d2eb"}},{"code_sha256_prefix":"0ef94faeaa86c9b2","entry":"load_and_merge_config","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"previous_version/Omni-Video1/finetune_model.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/previous_version/Omni-Video1/finetune_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ef94faeaa86c9b2"}},{"code_sha256_prefix":"36e61b3d6b43643e","entry":"load_qwen3vl_model_and_processor","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/vllm_model.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/vllm_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36e61b3d6b43643e"}},{"code_sha256_prefix":"d12b826fb528dd75","entry":"patchify","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/vae2_2.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/vae2_2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d12b826fb528dd75"}},{"code_sha256_prefix":"9208a89a1646609f","entry":"predict_target_video_caption","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/vllm_model.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/vllm_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9208a89a1646609f"}},{"code_sha256_prefix":"11ec08a7394a4fc5","entry":"rope_apply","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/model.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"11ec08a7394a4fc5"}},{"code_sha256_prefix":"4f942ef0b492ab0e","entry":"rope_params","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/model.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4f942ef0b492ab0e"}},{"code_sha256_prefix":"9937f1e6b1a8ab18","entry":"save_config_to_yaml","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"previous_version/Omni-Video1/finetune_model.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/previous_version/Omni-Video1/finetune_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9937f1e6b1a8ab18"}},{"code_sha256_prefix":"4d34354003484c3c","entry":"str2tuple","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"previous_version/Omni-Video1/finetune_model.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/previous_version/Omni-Video1/finetune_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d34354003484c3c"}},{"code_sha256_prefix":"e71b4f65ac349ba4","entry":"unpatchify","repo":"sais-fuxi/omni-video","repo_kind":"official","path":"omnivideo/modules/vae2_2.py","file_url":"https://github.com/sais-fuxi/omni-video/blob/HEAD/omnivideo/modules/vae2_2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e71b4f65ac349ba4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}