{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/conditional-image-to-video-generation-with","title":"Conditional Image-to-Video Generation with Latent Flow Diffusion Models","arxiv_id":"2303.13744","date":"2023-03-24","proceeding":"CVPR 2023 1","authors":["Haomiao Ni","Changhao Shi","Kai Li","Sharon X. Huang","Martin Renqiang Min"],"abstract":"Conditional image-to-video (cI2V) generation aims to synthesize a new plausible video starting from an image (e.g., a person's face) and a condition (e.g., an action class label like smile). The key challenge of the cI2V task lies in the simultaneous generation of realistic spatial appearance and temporal dynamics corresponding to the given image and condition. In this paper, we propose an approach for cI2V using novel latent flow diffusion models (LFDM) that synthesize an optical flow sequence in the latent space based on the given condition to warp the given image. Compared to previous direct-synthesis-based works, our proposed LFDM can better synthesize spatial details and temporal motion by fully utilizing the spatial content of the given image and warping it in the latent space according to the generated temporally-coherent flow. The training of LFDM consists of two separate stages: (1) an unsupervised learning stage to train a latent flow auto-encoder for spatial content generation, including a flow predictor to estimate latent flow between pairs of video frames, and (2) a conditional learning stage to train a 3D-UNet-based diffusion model (DM) for temporal latent flow generation. Unlike previous DMs operating in pixel space or latent feature space that couples spatial and temporal information, the DM in our LFDM only needs to learn a low-dimensional latent flow space for motion generation, thus being more computationally efficient. We conduct comprehensive experiments on multiple datasets, where LFDM consistently outperforms prior arts. Furthermore, we show that LFDM can be easily adapted to new domains by simply finetuning the image decoder. Our code is available at https://github.com/nihaomiao/CVPR23_LFDM.","url_abs":"https://arxiv.org/abs/2303.13744v1","url_pdf":"https://arxiv.org/pdf/2303.13744v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"conditional-image-to-video-generation-with","repo_url":"https://github.com/nihaomiao/cvpr23_lfdm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"tasks":[{"task_slug":"image-to-video","task_name":"Image to Video Generation"},{"task_slug":"motion-generation","task_name":"Motion Generation"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.13744","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.13744"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nihaomiao/cvpr23_lfdm","reach":{"status":"ok","spdx":"BSD-2-Clause"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/nihaomiao/CVPR23_LFDM","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"summary":{"ran_violates":4,"unverified":5},"by_repo_kind":{"official":{"samples":9,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d1ef6b8cb9a28a53","entry":"default","repo":"nihaomiao/CVPR23_LFDM","repo_kind":"official","path":"DM/modules/video_flow_diffusion.py","file_url":"https://github.com/nihaomiao/CVPR23_LFDM/blob/HEAD/DM/modules/video_flow_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d1ef6b8cb9a28a53"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"nihaomiao/CVPR23_LFDM","repo_kind":"official","path":"DM/modules/text.py","file_url":"https://github.com/nihaomiao/CVPR23_LFDM/blob/HEAD/DM/modules/text.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"608e364a9d2376a3","entry":"exists","repo":"nihaomiao/CVPR23_LFDM","repo_kind":"official","path":"DM/modules/video_flow_diffusion.py","file_url":"https://github.com/nihaomiao/CVPR23_LFDM/blob/HEAD/DM/modules/video_flow_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"608e364a9d2376a3"}},{"code_sha256_prefix":"ebc12c938e7a6b29","entry":"is_odd","repo":"nihaomiao/CVPR23_LFDM","repo_kind":"official","path":"DM/modules/video_flow_diffusion.py","file_url":"https://github.com/nihaomiao/CVPR23_LFDM/blob/HEAD/DM/modules/video_flow_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ebc12c938e7a6b29"}},{"code_sha256_prefix":"ac66a9775ac199a0","entry":"bert_embed","repo":"nihaomiao/CVPR23_LFDM","repo_kind":"official","path":"DM/modules/text.py","file_url":"https://github.com/nihaomiao/CVPR23_LFDM/blob/HEAD/DM/modules/text.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ac66a9775ac199a0"}},{"code_sha256_prefix":"910de59edc0ed1b0","entry":"fig2data","repo":"nihaomiao/CVPR23_LFDM","repo_kind":"official","path":"misc.py","file_url":"https://github.com/nihaomiao/CVPR23_LFDM/blob/HEAD/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"910de59edc0ed1b0"}},{"code_sha256_prefix":"d583c2d32880aade","entry":"grid2fig","repo":"nihaomiao/CVPR23_LFDM","repo_kind":"official","path":"misc.py","file_url":"https://github.com/nihaomiao/CVPR23_LFDM/blob/HEAD/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d583c2d32880aade"}},{"code_sha256_prefix":"fc24f5a835af0997","entry":"sample_img","repo":"nihaomiao/CVPR23_LFDM","repo_kind":"official","path":"DM/train_video_flow_diffusion_mhad.py","file_url":"https://github.com/nihaomiao/CVPR23_LFDM/blob/HEAD/DM/train_video_flow_diffusion_mhad.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"fc24f5a835af0997"}},{"code_sha256_prefix":"03dfcc33eeaff851","entry":"tokenize","repo":"nihaomiao/CVPR23_LFDM","repo_kind":"official","path":"DM/modules/text.py","file_url":"https://github.com/nihaomiao/CVPR23_LFDM/blob/HEAD/DM/modules/text.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"03dfcc33eeaff851"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}