{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/videocrafter2-overcoming-data-limitations-for","title":"VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models","arxiv_id":"2401.09047","date":"2024-01-17","proceeding":"CVPR 2024 1","authors":["Haoxin Chen","Yong Zhang","Xiaodong Cun","Menghan Xia","Xintao Wang","Chao Weng","Ying Shan"],"abstract":"Text-to-video generation aims to produce a video based on a given prompt. Recently, several commercial video models have been able to generate plausible videos with minimal noise, excellent details, and high aesthetic scores. However, these models rely on large-scale, well-filtered, high-quality videos that are not accessible to the community. Many existing research works, which train models using the low-quality WebVid-10M dataset, struggle to generate high-quality videos because the models are optimized to fit WebVid-10M. In this work, we explore the training scheme of video models extended from Stable Diffusion and investigate the feasibility of leveraging low-quality videos and synthesized high-quality images to obtain a high-quality video model. We first analyze the connection between the spatial and temporal modules of video models and the distribution shift to low-quality videos. We observe that full training of all modules results in a stronger coupling between spatial and temporal modules than only training temporal modules. Based on this stronger coupling, we shift the distribution to higher quality without motion degradation by finetuning spatial modules with high-quality images, resulting in a generic high-quality video model. Evaluations are conducted to demonstrate the superiority of the proposed method, particularly in picture quality, motion, and concept composition.","url_abs":"https://arxiv.org/abs/2401.09047v1","url_pdf":"https://arxiv.org/pdf/2401.09047v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"videocrafter2-overcoming-data-limitations-for","repo_url":"https://github.com/ailab-cvc/videocrafter","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"videocrafter2-overcoming-data-limitations-for","repo_url":"https://github.com/videocrafter/videocrafter","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"text-to-video-generation","task_name":"Text-to-Video Generation"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/text-to-video-generation-on-evalcrafter-text","task":"Text-to-Video Generation","dataset":"EvalCrafter Text-to-Video (ECTV) Dataset","model":"VideoCrafter2","rank_in_archive_order":1,"of":5,"metrics":{"Motion Quality":"63.98","Temporal Consistency":"61.46","Text-to-Video Alignment":"63.16","Total Score":"243","Visual Quality":"54.82"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2401.09047","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.09047"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ailab-cvc/videocrafter","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/videocrafter/videocrafter","reach":null}],"summary":{"ran":4,"ran_draft_wrong":3,"ran_honours":3,"ran_violates":3,"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":16,"ran":12,"repositories":1},"listed":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":21,"samples":[{"code_sha256_prefix":"7c6e76f1f7ce1dbc","entry":"DiagonalGaussianDistribution","repo":"videocrafter/videocrafter","repo_kind":"listed","path":"lvdm/models/ddpm3d.py","file_url":"https://github.com/videocrafter/videocrafter/blob/HEAD/lvdm/models/ddpm3d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7c6e76f1f7ce1dbc"}},{"code_sha256_prefix":"fe5dd5258046898c","entry":"always","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/modules/x_transformer.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/modules/x_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe5dd5258046898c"}},{"code_sha256_prefix":"19f538291be81c23","entry":"check_istarget","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"19f538291be81c23"}},{"code_sha256_prefix":"613acd72145a95f2","entry":"count_params","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"613acd72145a95f2"}},{"code_sha256_prefix":"424012cb37b31172","entry":"default","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/modules/x_transformer.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/modules/x_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"424012cb37b31172"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/basics.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/basics.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/modules/x_transformer.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/modules/x_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"a15cfd7932844ef9","entry":"extract_into_tensor","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/common.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/common.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a15cfd7932844ef9"}},{"code_sha256_prefix":"c984209a2c66afbe","entry":"make_beta_schedule","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/models/utils_diffusion.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/models/utils_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c984209a2c66afbe"}},{"code_sha256_prefix":"0ea4e960ea54514c","entry":"make_ddim_timesteps","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/models/utils_diffusion.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/models/utils_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ea4e960ea54514c"}},{"code_sha256_prefix":"f791d43d666fedd1","entry":"noise_like","repo":"videocrafter/videocrafter","repo_kind":"listed","path":"lvdm/models/ddpm3d.py","file_url":"https://github.com/videocrafter/videocrafter/blob/HEAD/lvdm/models/ddpm3d.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f791d43d666fedd1"}},{"code_sha256_prefix":"17eb0a31c91b671c","entry":"normal_kl","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/distributions.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/distributions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17eb0a31c91b671c"}},{"code_sha256_prefix":"2203f9d1be3dc36c","entry":"timestep_embedding","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/models/utils_diffusion.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/models/utils_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2203f9d1be3dc36c"}},{"code_sha256_prefix":"129b804760b3115f","entry":"zero_module","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/basics.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/basics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"129b804760b3115f"}},{"code_sha256_prefix":"21a8fb71de781ce8","entry":"DDPM","repo":"videocrafter/videocrafter","repo_kind":"listed","path":"lvdm/models/ddpm3d.py","file_url":"https://github.com/videocrafter/videocrafter/blob/HEAD/lvdm/models/ddpm3d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"21a8fb71de781ce8"}},{"code_sha256_prefix":"5c9c5717118146d8","entry":"DiffusionWrapper","repo":"videocrafter/videocrafter","repo_kind":"listed","path":"lvdm/models/ddpm3d.py","file_url":"https://github.com/videocrafter/videocrafter/blob/HEAD/lvdm/models/ddpm3d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5c9c5717118146d8"}},{"code_sha256_prefix":"c4747d069e654ef3","entry":"LatentDiffusion","repo":"videocrafter/videocrafter","repo_kind":"listed","path":"lvdm/models/ddpm3d.py","file_url":"https://github.com/videocrafter/videocrafter/blob/HEAD/lvdm/models/ddpm3d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c4747d069e654ef3"}},{"code_sha256_prefix":"32af1c12fd826ef2","entry":"autocast","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/common.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32af1c12fd826ef2"}},{"code_sha256_prefix":"b72d8eb89fd75383","entry":"gather_data","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/common.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b72d8eb89fd75383"}},{"code_sha256_prefix":"ef52d142b2c30ac2","entry":"instantiate_from_config","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef52d142b2c30ac2"}},{"code_sha256_prefix":"9136b5f9b735334b","entry":"scale_module","repo":"ailab-cvc/videocrafter","repo_kind":"official","path":"lvdm/basics.py","file_url":"https://github.com/ailab-cvc/videocrafter/blob/HEAD/lvdm/basics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9136b5f9b735334b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}