{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/align-your-latents-high-resolution-video","title":"Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models","arxiv_id":"2304.08818","date":"2023-04-18","proceeding":"CVPR 2023 1","authors":["Andreas Blattmann","Robin Rombach","Huan Ling","Tim Dockhorn","Seung Wook Kim","Sanja Fidler","Karsten Kreis"],"abstract":"Latent Diffusion Models (LDMs) enable high-quality image synthesis while avoiding excessive compute demands by training a diffusion model in a compressed lower-dimensional latent space. Here, we apply the LDM paradigm to high-resolution video generation, a particularly resource-intensive task. We first pre-train an LDM on images only; then, we turn the image generator into a video generator by introducing a temporal dimension to the latent space diffusion model and fine-tuning on encoded image sequences, i.e., videos. Similarly, we temporally align diffusion model upsamplers, turning them into temporally consistent video super resolution models. We focus on two relevant real-world applications: Simulation of in-the-wild driving data and creative content creation with text-to-video modeling. In particular, we validate our Video LDM on real driving videos of resolution 512 x 1024, achieving state-of-the-art performance. Furthermore, our approach can easily leverage off-the-shelf pre-trained image LDMs, as we only need to train a temporal alignment model in that case. Doing so, we turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to 1280 x 2048. We show that the temporal layers trained in this way generalize to different fine-tuned text-to-image LDMs. Utilizing this property, we show the first results for personalized text-to-video generation, opening exciting directions for future content creation. Project page: https://research.nvidia.com/labs/toronto-ai/VideoLDM/","url_abs":"https://arxiv.org/abs/2304.08818v2","url_pdf":"https://arxiv.org/pdf/2304.08818v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"align-your-latents-high-resolution-video","repo_url":"https://github.com/stability-ai/generative-models","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"align-your-latents-high-resolution-video","repo_url":"https://github.com/ai-forever/kandinskyvideo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"align-your-latents-high-resolution-video","repo_url":"https://github.com/gongzix/neuroclips","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"align-your-latents-high-resolution-video","repo_url":"https://github.com/srpkdyy/VideoLDM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"super-resolution","task_name":"Super-Resolution"},{"task_slug":"text-to-video-generation","task_name":"Text-to-Video Generation"},{"task_slug":"video-generation","task_name":"Video Generation"},{"task_slug":"video-super-resolution","task_name":"Video Super-Resolution"},{"task_slug":"high","task_name":"Vocal Bursts Intensity Prediction"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/text-to-video-generation-on-msr-vtt","task":"Text-to-Video Generation","dataset":"MSR-VTT","model":"Video LDM","rank_in_archive_order":13,"of":18,"metrics":{"CLIPSIM":"0.2929"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-video-generation-on-msr-vtt","task":"Text-to-Video Generation","dataset":"MSR-VTT","model":"CogVideo (Chinese)","rank_in_archive_order":16,"of":18,"metrics":{"CLIP-FID":"24.78","CLIPSIM":"0.2614"},"uses_additional_data":false},{"leaderboard":"/sota/text-to-video-generation-on-ucf-101","task":"Text-to-Video Generation","dataset":"UCF-101","model":"Video LDM (Zero-shot, 320x512)","rank_in_archive_order":9,"of":10,"metrics":{"FVD16":"550.61"},"uses_additional_data":false},{"leaderboard":"/sota/video-generation-on-ucf-101","task":"Video Generation","dataset":"UCF-101","model":"Video LDM (320x512, text-conditional)","rank_in_archive_order":38,"of":48,"metrics":{"FVD16":"550.61","Inception Score":"33.45"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2304.08818","atlas_url":"https://app.syntology.ai/?focus=2304.08818","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.08818"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ai-forever/kandinskyvideo","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gongzix/neuroclips","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/srpkdyy/VideoLDM","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stability-ai/generative-models","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/NVlabs/long-video-gan","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":6,"ran_violates":3,"ran":7,"ran_honours":2,"unverified":8},"by_repo_kind":{"listed":{"samples":13,"ran":6,"repositories":1},"found_in_text":{"samples":12,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"fe5dd5258046898c","entry":"always","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"ldm/modules/x_transformer.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/ldm/modules/x_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe5dd5258046898c"}},{"code_sha256_prefix":"8241c0562bc710fd","entry":"chunk","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"8241c0562bc710fd"}},{"code_sha256_prefix":"424012cb37b31172","entry":"default","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"ldm/modules/attention.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/ldm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"424012cb37b31172"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"ldm/models/diffusion/classifier.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/ldm/models/diffusion/classifier.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"2d8424245424b7c2","entry":"exist","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/model/utils.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/model/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d8424245424b7c2"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"ldm/modules/attention.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/ldm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"ffd1b5056434604b","entry":"freeze","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/utils.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ffd1b5056434604b"}},{"code_sha256_prefix":"d5726f9e3a5684c3","entry":"get_bert_token_for_string","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"ldm/modules/embedding_manager.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/ldm/modules/embedding_manager.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d5726f9e3a5684c3"}},{"code_sha256_prefix":"41c6eb3a0aa3eb46","entry":"get_clip_token_for_string","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"ldm/modules/embedding_manager.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/ldm/modules/embedding_manager.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41c6eb3a0aa3eb46"}},{"code_sha256_prefix":"e466f56ae7a8bea6","entry":"get_clip_token_for_string","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"merge_embeddings.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/merge_embeddings.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e466f56ae7a8bea6"}},{"code_sha256_prefix":"71c2df28a1008d03","entry":"get_embedding_for_clip_token","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"ldm/modules/embedding_manager.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/ldm/modules/embedding_manager.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71c2df28a1008d03"}},{"code_sha256_prefix":"cb1a19e61d70046b","entry":"get_placeholder_loop","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"merge_embeddings.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/merge_embeddings.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb1a19e61d70046b"}},{"code_sha256_prefix":"388d5db2e1f7c3e9","entry":"load_conf","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/utils.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"388d5db2e1f7c3e9"}},{"code_sha256_prefix":"3137073275f8c21a","entry":"nonlinearity","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/movq.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/movq.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3137073275f8c21a"}},{"code_sha256_prefix":"5da5a9e348708c11","entry":"set_default_item","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/model/utils.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/model/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5da5a9e348708c11"}},{"code_sha256_prefix":"7831948ab4cf9cc6","entry":"unfreeze","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/utils.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7831948ab4cf9cc6"}},{"code_sha256_prefix":"d48d8354986e3b0e","entry":"uniform_on_device","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"ldm/models/diffusion/ddpm.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/ldm/models/diffusion/ddpm.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d48d8354986e3b0e"}},{"code_sha256_prefix":"9a299fe5ae09e407","entry":"uniq","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"ldm/modules/attention.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/ldm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a299fe5ae09e407"}},{"code_sha256_prefix":"8e576cbf9f53dd37","entry":"Normalize","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/movq.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/movq.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8e576cbf9f53dd37"}},{"code_sha256_prefix":"91610d6e2f2a67b4","entry":"get_bert_token_for_string","repo":"XavierXiao/Dreambooth-Stable-Diffusion","repo_kind":"found_in_text","path":"merge_embeddings.py","file_url":"https://github.com/XavierXiao/Dreambooth-Stable-Diffusion/blob/HEAD/merge_embeddings.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91610d6e2f2a67b4"}},{"code_sha256_prefix":"0e4409aa17549faa","entry":"get_condition_encoder","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/condition_encoders.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/condition_encoders.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e4409aa17549faa"}},{"code_sha256_prefix":"887b76ef7e5c376f","entry":"get_sampler","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/model/diffusion.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/model/diffusion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"887b76ef7e5c376f"}},{"code_sha256_prefix":"ed0f76c70dc89762","entry":"get_unet","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/model/unet.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/model/unet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed0f76c70dc89762"}},{"code_sha256_prefix":"32876ae4d79d1787","entry":"get_unet","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/model/unet_interpolation.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/model/unet_interpolation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32876ae4d79d1787"}},{"code_sha256_prefix":"af44aa76c2918b6b","entry":"get_vae","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/movq.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/movq.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af44aa76c2918b6b"}},{"code_sha256_prefix":"325f8f5ac2f1d3a6","entry":"set_default_layer","repo":"ai-forever/kandinskyvideo","repo_kind":"listed","path":"kandinsky_video/model/utils.py","file_url":"https://github.com/ai-forever/kandinskyvideo/blob/HEAD/kandinsky_video/model/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"325f8f5ac2f1d3a6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}