{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/omnitokenizer-a-joint-image-video-tokenizer","title":"OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation","arxiv_id":"2406.09399","date":"2024-06-13","proceeding":null,"authors":["Junke Wang","Yi Jiang","Zehuan Yuan","Binyue Peng","Zuxuan Wu","Yu-Gang Jiang"],"abstract":"Tokenizer, serving as a translator to map the intricate visual data into a compact latent space, lies at the core of visual generative models. Based on the finding that existing tokenizers are tailored to image or video inputs, this paper presents OmniTokenizer, a transformer-based tokenizer for joint image and video tokenization. OmniTokenizer is designed with a spatial-temporal decoupled architecture, which integrates window and causal attention for spatial and temporal modeling. To exploit the complementary nature of image and video data, we further propose a progressive training strategy, where OmniTokenizer is first trained on image data on a fixed resolution to develop the spatial encoding capacity and then jointly trained on image and video data on multiple resolutions to learn the temporal dynamics. OmniTokenizer, for the first time, handles both image and video inputs within a unified framework and proves the possibility of realizing their synergy. Extensive experiments demonstrate that OmniTokenizer achieves state-of-the-art (SOTA) reconstruction performance on various image and video datasets, e.g., 1.11 reconstruction FID on ImageNet and 42 reconstruction FVD on UCF-101, beating the previous SOTA methods by 13% and 26%, respectively. Additionally, we also show that when integrated with OmniTokenizer, both language model-based approaches and diffusion models can realize advanced visual synthesis performance, underscoring the superiority and versatility of our method. Code is available at https://github.com/FoundationVision/OmniTokenizer.","url_abs":"https://arxiv.org/abs/2406.09399v1","url_pdf":"https://arxiv.org/pdf/2406.09399v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"omnitokenizer-a-joint-image-video-tokenizer","repo_url":"https://github.com/foundationvision/omnitokenizer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"video-generation","task_name":"Video Generation"},{"task_slug":"video-prediction","task_name":"Video Prediction"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-generation-on-ucf-101","task":"Video Generation","dataset":"UCF-101","model":"OmniTokenizer-AR","rank_in_archive_order":15,"of":48,"metrics":{"FVD16":"191"},"uses_additional_data":false},{"leaderboard":"/sota/video-prediction-on-kinetics-600-12-frames","task":"Video Prediction","dataset":"Kinetics-600 12 frames, 64x64","model":"OmniTokenizer-AR","rank_in_archive_order":11,"of":16,"metrics":{"FVD":"32.9"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.09399","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.09399"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/foundationvision/omnitokenizer","reach":null}],"summary":{"ran_violates":1,"ran_fixture":2,"ran_draft_wrong":1,"ran_honours":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"856d86b39c6005b4","entry":"divisible_by","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"856d86b39c6005b4"}},{"code_sha256_prefix":"07bca1f7d65cae68","entry":"calculate_fvd","repo":"foundationvision/omnitokenizer","repo_kind":"official","path":"evaluation/common_metrics_on_video_quality/calculate_fvd.py","file_url":"https://github.com/foundationvision/omnitokenizer/blob/HEAD/evaluation/common_metrics_on_video_quality/calculate_fvd.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07bca1f7d65cae68"}},{"code_sha256_prefix":"3cf68464859086b1","entry":"logits_laplace","repo":"foundationvision/omnitokenizer","repo_kind":"official","path":"OmniTokenizer/omnitokenizer.py","file_url":"https://github.com/foundationvision/omnitokenizer/blob/HEAD/OmniTokenizer/omnitokenizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3cf68464859086b1"}},{"code_sha256_prefix":"6d5ab71931a90677","entry":"pair","repo":"foundationvision/omnitokenizer","repo_kind":"official","path":"OmniTokenizer/omnitokenizer.py","file_url":"https://github.com/foundationvision/omnitokenizer/blob/HEAD/OmniTokenizer/omnitokenizer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d5ab71931a90677"}},{"code_sha256_prefix":"33b7df59642bf5f9","entry":"trans","repo":"foundationvision/omnitokenizer","repo_kind":"official","path":"evaluation/common_metrics_on_video_quality/calculate_fvd.py","file_url":"https://github.com/foundationvision/omnitokenizer/blob/HEAD/evaluation/common_metrics_on_video_quality/calculate_fvd.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33b7df59642bf5f9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}