{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dynamicrafter-animating-open-domain-images","title":"DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors","arxiv_id":"2310.12190","date":"2023-10-18","proceeding":null,"authors":["Jinbo Xing","Menghan Xia","Yong Zhang","Haoxin Chen","Wangbo Yu","Hanyuan Liu","Xintao Wang","Tien-Tsin Wong","Ying Shan"],"abstract":"Animating a still image offers an engaging visual experience. Traditional image animation techniques mainly focus on animating natural scenes with stochastic dynamics (e.g. clouds and fluid) or domain-specific motions (e.g. human hair or body motions), and thus limits their applicability to more general visual content. To overcome this limitation, we explore the synthesis of dynamic content for open-domain images, converting them into animated videos. The key idea is to utilize the motion prior of text-to-video diffusion models by incorporating the image into the generative process as guidance. Given an image, we first project it into a text-aligned rich context representation space using a query transformer, which facilitates the video model to digest the image content in a compatible fashion. However, some visual details still struggle to be preserved in the resultant videos. To supplement with more precise image information, we further feed the full image to the diffusion model by concatenating it with the initial noises. Experimental results show that our proposed method can produce visually convincing and more logical & natural motions, as well as higher conformity to the input image. Comparative evaluation demonstrates the notable superiority of our approach over existing competitors.","url_abs":"https://arxiv.org/abs/2310.12190v2","url_pdf":"https://arxiv.org/pdf/2310.12190v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dynamicrafter-animating-open-domain-images","repo_url":"https://github.com/Doubiiu/DynamiCrafter","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"dynamicrafter-animating-open-domain-images","repo_url":"https://github.com/soCzech/ShowHowTo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-animation","task_name":"Image Animation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.12190","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.12190"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/soCzech/ShowHowTo","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Doubiiu/DynamiCrafter","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"ran_violates":3,"ran_fixture":2,"ran_honours":1,"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":13,"ran":9,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fe5dd5258046898c","entry":"always","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/modules/x_transformer.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/modules/x_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe5dd5258046898c"}},{"code_sha256_prefix":"424012cb37b31172","entry":"default","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/modules/x_transformer.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/modules/x_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"424012cb37b31172"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/basics.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/basics.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/modules/x_transformer.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/modules/x_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"a15cfd7932844ef9","entry":"extract_into_tensor","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/common.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/common.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a15cfd7932844ef9"}},{"code_sha256_prefix":"41bf0ab789ce9971","entry":"get_latent_z","repo":"soCzech/ShowHowTo","repo_kind":"listed","path":"predict.py","file_url":"https://github.com/soCzech/ShowHowTo/blob/HEAD/predict.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41bf0ab789ce9971"}},{"code_sha256_prefix":"c984209a2c66afbe","entry":"make_beta_schedule","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/models/utils_diffusion.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/models/utils_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c984209a2c66afbe"}},{"code_sha256_prefix":"17eb0a31c91b671c","entry":"normal_kl","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/distributions.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/distributions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17eb0a31c91b671c"}},{"code_sha256_prefix":"7e83c65f054fb972","entry":"timestep_embedding","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/models/utils_diffusion.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/models/utils_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e83c65f054fb972"}},{"code_sha256_prefix":"129b804760b3115f","entry":"zero_module","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/basics.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/basics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"129b804760b3115f"}},{"code_sha256_prefix":"32af1c12fd826ef2","entry":"autocast","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/common.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32af1c12fd826ef2"}},{"code_sha256_prefix":"b72d8eb89fd75383","entry":"gather_data","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/common.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b72d8eb89fd75383"}},{"code_sha256_prefix":"662fc3ac21ada8a7","entry":"make_ddim_timesteps","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/models/utils_diffusion.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/models/utils_diffusion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"662fc3ac21ada8a7"}},{"code_sha256_prefix":"9136b5f9b735334b","entry":"scale_module","repo":"Doubiiu/DynamiCrafter","repo_kind":"official","path":"lvdm/basics.py","file_url":"https://github.com/Doubiiu/DynamiCrafter/blob/HEAD/lvdm/basics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9136b5f9b735334b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}