{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/motionclone-training-free-motion-cloning-for","title":"MotionClone: Training-Free Motion Cloning for Controllable Video Generation","arxiv_id":"2406.05338","date":"2024-06-08","proceeding":null,"authors":["Pengyang Ling","Jiazi Bu","Pan Zhang","Xiaoyi Dong","Yuhang Zang","Tong Wu","Huaian Chen","Jiaqi Wang","Yi Jin"],"abstract":"Motion-based controllable video generation offers the potential for creating captivating visual content. Existing methods typically necessitate model training to encode particular motion cues or incorporate fine-tuning to inject certain motion patterns, resulting in limited flexibility and generalization. In this work, we propose MotionClone, a training-free framework that enables motion cloning from reference videos to versatile motion-controlled video generation, including text-to-video and image-to-video. Based on the observation that the dominant components in temporal-attention maps drive motion synthesis, while the rest mainly capture noisy or very subtle motions, MotionClone utilizes sparse temporal attention weights as motion representations for motion guidance, facilitating diverse motion transfer across varying scenarios. Meanwhile, MotionClone allows for the direct extraction of motion representation through a single denoising step, bypassing the cumbersome inversion processes and thus promoting both efficiency and flexibility. Extensive experiments demonstrate that MotionClone exhibits proficiency in both global camera motion and local object motion, with notable superiority in terms of motion fidelity, textual alignment, and temporal consistency.","url_abs":"https://arxiv.org/abs/2406.05338v6","url_pdf":"https://arxiv.org/pdf/2406.05338v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"motionclone-training-free-motion-cloning-for","repo_url":"https://github.com/bujiazi/motionclone","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"motionclone-training-free-motion-cloning-for","repo_url":"https://github.com/lpengyang/motionclone","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"motion-generation","task_name":"Motion Generation"},{"task_slug":"motion-synthesis","task_name":"Motion Synthesis"},{"task_slug":"text-to-video-generation","task_name":"Text-to-Video Generation"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.05338","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.05338"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lpengyang/motionclone","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bujiazi/motionclone","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":9},"by_repo_kind":{"official":{"samples":15,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"a9ca93a52881cc27","entry":"add_noise","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/utils/motionclone_functions.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/utils/motionclone_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9ca93a52881cc27"}},{"code_sha256_prefix":"bd248479929335fd","entry":"conv_forward","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/utils/conv_layer.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/utils/conv_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd248479929335fd"}},{"code_sha256_prefix":"fecbe010c76c8b54","entry":"get_conv_feat","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/utils/conv_layer.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/utils/conv_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fecbe010c76c8b54"}},{"code_sha256_prefix":"f3c0f6e327171982","entry":"prep_unet_conv","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/utils/conv_layer.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/utils/conv_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3c0f6e327171982"}},{"code_sha256_prefix":"bf5971f2cfe5d88f","entry":"sample_video","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/utils/motionclone_functions.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/utils/motionclone_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bf5971f2cfe5d88f"}},{"code_sha256_prefix":"4719c763c53be3fe","entry":"zero_module","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/models/motion_module.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/models/motion_module.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4719c763c53be3fe"}},{"code_sha256_prefix":"67f3b5cb045eae24","entry":"compute_temp_loss","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/utils/motionclone_functions.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/utils/motionclone_functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67f3b5cb045eae24"}},{"code_sha256_prefix":"9736e3792b57e25c","entry":"convert_lora","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/utils/convert_lora_safetensor_to_diffusers.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/utils/convert_lora_safetensor_to_diffusers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9736e3792b57e25c"}},{"code_sha256_prefix":"47ccb78720fa43a4","entry":"get_down_block","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/models/unet_blocks.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/models/unet_blocks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"47ccb78720fa43a4"}},{"code_sha256_prefix":"17bb6aea1323b38f","entry":"get_up_block","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/models/unet_blocks.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/models/unet_blocks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17bb6aea1323b38f"}},{"code_sha256_prefix":"c43325ffa267c1b0","entry":"load_diffusers_lora","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/utils/convert_lora_safetensor_to_diffusers.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/utils/convert_lora_safetensor_to_diffusers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c43325ffa267c1b0"}},{"code_sha256_prefix":"dfb052a0af48540f","entry":"renew_resnet_paths","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/utils/convert_from_ckpt.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/utils/convert_from_ckpt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfb052a0af48540f"}},{"code_sha256_prefix":"c5ec9c13e456a046","entry":"renew_vae_resnet_paths","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/utils/convert_from_ckpt.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/utils/convert_from_ckpt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c5ec9c13e456a046"}},{"code_sha256_prefix":"cea0bc82e0c96896","entry":"shave_segments","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/utils/convert_from_ckpt.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/utils/convert_from_ckpt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cea0bc82e0c96896"}},{"code_sha256_prefix":"da94debb8019ad46","entry":"zero_module","repo":"bujiazi/motionclone","repo_kind":"official","path":"motionclone/models/sparse_controlnet.py","file_url":"https://github.com/bujiazi/motionclone/blob/HEAD/motionclone/models/sparse_controlnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da94debb8019ad46"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}