{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cogvideox-text-to-video-diffusion-models-with","title":"CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer","arxiv_id":"2408.06072","date":"2024-08-12","proceeding":null,"authors":["Zhuoyi Yang","Jiayan Teng","Wendi Zheng","Ming Ding","Shiyu Huang","Jiazheng Xu","Yuanming Yang","Wenyi Hong","Xiaohan Zhang","Guanyu Feng","Da Yin","Yuxuan Zhang","Weihan Wang","Yean Cheng","Bin Xu","Xiaotao Gu","Yuxiao Dong","Jie Tang"],"abstract":"We present CogVideoX, a large-scale text-to-video generation model based on diffusion transformer, which can generate 10-second continuous videos aligned with text prompt, with a frame rate of 16 fps and resolution of 768 * 1360 pixels. Previous video generation models often had limited movement and short durations, and is difficult to generate videos with coherent narratives based on text. We propose several designs to address these issues. First, we propose a 3D Variational Autoencoder (VAE) to compress videos along both spatial and temporal dimensions, to improve both compression rate and video fidelity. Second, to improve the text-video alignment, we propose an expert transformer with the expert adaptive LayerNorm to facilitate the deep fusion between the two modalities. Third, by employing a progressive training and multi-resolution frame pack technique, CogVideoX is adept at producing coherent, long-duration, different shape videos characterized by significant motions. In addition, we develop an effective text-video data processing pipeline that includes various data preprocessing strategies and a video captioning method, greatly contributing to the generation quality and semantic alignment. Results show that CogVideoX demonstrates state-of-the-art performance across both multiple machine metrics and human evaluations. The model weight of both 3D Causal VAE, Video caption model and CogVideoX are publicly available at https://github.com/THUDM/CogVideo.","url_abs":"https://arxiv.org/abs/2408.06072v3","url_pdf":"https://arxiv.org/pdf/2408.06072v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cogvideox-text-to-video-diffusion-models-with","repo_url":"https://github.com/thudm/cogvideo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"cogvideox-text-to-video-diffusion-models-with","repo_url":"https://github.com/gulucaptain/dynamictrl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"text-to-video-generation","task_name":"Text-to-Video Generation"},{"task_slug":"video-alignment","task_name":"Video Alignment"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.06072","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.06072"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thudm/cogvideo","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gulucaptain/dynamictrl","reach":{"status":"ok"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c5d7e2dfc17a2849","entry":"bytes_to_gigabytes","repo":"thudm/cogvideo","repo_kind":"official","path":"finetune/utils/memory_utils.py","file_url":"https://github.com/thudm/cogvideo/blob/HEAD/finetune/utils/memory_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c5d7e2dfc17a2849"}},{"code_sha256_prefix":"b79185cd9f8337d1","entry":"find_files","repo":"thudm/cogvideo","repo_kind":"official","path":"finetune/utils/file_utils.py","file_url":"https://github.com/thudm/cogvideo/blob/HEAD/finetune/utils/file_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b79185cd9f8337d1"}},{"code_sha256_prefix":"4b69651f45d261b5","entry":"load_images","repo":"thudm/cogvideo","repo_kind":"official","path":"finetune/datasets/utils.py","file_url":"https://github.com/thudm/cogvideo/blob/HEAD/finetune/datasets/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b69651f45d261b5"}},{"code_sha256_prefix":"fad222ebf1926ea1","entry":"load_prompts","repo":"thudm/cogvideo","repo_kind":"official","path":"finetune/datasets/utils.py","file_url":"https://github.com/thudm/cogvideo/blob/HEAD/finetune/datasets/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fad222ebf1926ea1"}},{"code_sha256_prefix":"2fb1df079657be8e","entry":"load_videos","repo":"thudm/cogvideo","repo_kind":"official","path":"finetune/datasets/utils.py","file_url":"https://github.com/thudm/cogvideo/blob/HEAD/finetune/datasets/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2fb1df079657be8e"}},{"code_sha256_prefix":"9a2e1927af8da30a","entry":"make_contiguous","repo":"thudm/cogvideo","repo_kind":"official","path":"finetune/utils/memory_utils.py","file_url":"https://github.com/thudm/cogvideo/blob/HEAD/finetune/utils/memory_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a2e1927af8da30a"}},{"code_sha256_prefix":"4305e4a29601183d","entry":"string_to_filename","repo":"thudm/cogvideo","repo_kind":"official","path":"finetune/utils/file_utils.py","file_url":"https://github.com/thudm/cogvideo/blob/HEAD/finetune/utils/file_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4305e4a29601183d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}