{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/motioncraft-physics-based-zero-shot-video","title":"MotionCraft: Physics-based Zero-Shot Video Generation","arxiv_id":"2405.13557","date":"2024-05-22","proceeding":null,"authors":["Luca Savant Aira","Antonio Montanaro","Emanuele Aiello","Diego Valsesia","Enrico Magli"],"abstract":"Generating videos with realistic and physically plausible motion is one of the main recent challenges in computer vision. While diffusion models are achieving compelling results in image generation, video diffusion models are limited by heavy training and huge models, resulting in videos that are still biased to the training dataset. In this work we propose MotionCraft, a new zero-shot video generator to craft physics-based and realistic videos. MotionCraft is able to warp the noise latent space of an image diffusion model, such as Stable Diffusion, by applying an optical flow derived from a physics simulation. We show that warping the noise latent space results in coherent application of the desired motion while allowing the model to generate missing elements consistent with the scene evolution, which would otherwise result in artefacts or missing content if the flow was applied in the pixel space. We compare our method with the state-of-the-art Text2Video-Zero reporting qualitative and quantitative improvements, demonstrating the effectiveness of our approach to generate videos with finely-prescribed complex motion dynamics. Project page: https://mezzelfo.github.io/MotionCraft/","url_abs":"https://arxiv.org/abs/2405.13557v2","url_pdf":"https://arxiv.org/pdf/2405.13557v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"motioncraft-physics-based-zero-shot-video","repo_url":"https://github.com/mezzelfo/MotionCraft","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"missing-elements","task_name":"Missing Elements"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.13557","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.13557"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mezzelfo/MotionCraft","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"52fe8c867c821af3","entry":"get_all_schedulers","repo":"mezzelfo/MotionCraft","repo_kind":"official","path":"src/DiffusersUtils/myschedulers.py","file_url":"https://github.com/mezzelfo/MotionCraft/blob/HEAD/src/DiffusersUtils/myschedulers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52fe8c867c821af3"}},{"code_sha256_prefix":"de5f24b69443f083","entry":"get_attention_processor","repo":"mezzelfo/MotionCraft","repo_kind":"official","path":"src/DiffusersUtils/attention_processor.py","file_url":"https://github.com/mezzelfo/MotionCraft/blob/HEAD/src/DiffusersUtils/attention_processor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de5f24b69443f083"}},{"code_sha256_prefix":"ce3b1401f41799f8","entry":"get_attention_processor_from_pattern","repo":"mezzelfo/MotionCraft","repo_kind":"official","path":"src/DiffusersUtils/attention_processor.py","file_url":"https://github.com/mezzelfo/MotionCraft/blob/HEAD/src/DiffusersUtils/attention_processor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce3b1401f41799f8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}