{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mofa-video-controllable-image-animation-via","title":"MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model","arxiv_id":"2405.20222","date":"2024-05-30","proceeding":null,"authors":["Muyao Niu","Xiaodong Cun","Xintao Wang","Yong Zhang","Ying Shan","Yinqiang Zheng"],"abstract":"We present MOFA-Video, an advanced controllable image animation method that generates video from the given image using various additional controllable signals (such as human landmarks reference, manual trajectories, and another even provided video) or their combinations. This is different from previous methods which only can work on a specific motion domain or show weak control abilities with diffusion prior. To achieve our goal, we design several domain-aware motion field adapters (\\ie, MOFA-Adapters) to control the generated motions in the video generation pipeline. For MOFA-Adapters, we consider the temporal motion consistency of the video and generate the dense motion flow from the given sparse control conditions first, and then, the multi-scale features of the given image are wrapped as a guided feature for stable video diffusion generation. We naively train two motion adapters for the manual trajectories and the human landmarks individually since they both contain sparse information about the control. After training, the MOFA-Adapters in different domains can also work together for more controllable video generation. Project Page: https://myniuuu.github.io/MOFA_Video/","url_abs":"https://arxiv.org/abs/2405.20222v3","url_pdf":"https://arxiv.org/pdf/2405.20222v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mofa-video-controllable-image-animation-via","repo_url":"https://github.com/myniuuu/mofa-video","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"image-animation","task_name":"Image Animation"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.20222","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.20222"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/myniuuu/mofa-video","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/MyNiuuu/MOFA-Video","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"60212a22f2807648","entry":"draw_landmarks_cv2","repo":"MyNiuuu/MOFA-Video","repo_kind":"official","path":"MOFA-Video-Hybrid/run_gradio_audio_driven.py","file_url":"https://github.com/MyNiuuu/MOFA-Video/blob/HEAD/MOFA-Video-Hybrid/run_gradio_audio_driven.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"60212a22f2807648"}},{"code_sha256_prefix":"c03503a4357a244f","entry":"resize_and_crop","repo":"MyNiuuu/MOFA-Video","repo_kind":"official","path":"MOFA-Video-Keypoint/inference_opendomain.py","file_url":"https://github.com/MyNiuuu/MOFA-Video/blob/HEAD/MOFA-Video-Keypoint/inference_opendomain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c03503a4357a244f"}},{"code_sha256_prefix":"eda768a3f1bbc3fa","entry":"sample_optical_flow","repo":"MyNiuuu/MOFA-Video","repo_kind":"official","path":"MOFA-Video-Hybrid/run_gradio_audio_driven.py","file_url":"https://github.com/MyNiuuu/MOFA-Video/blob/HEAD/MOFA-Video-Hybrid/run_gradio_audio_driven.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"eda768a3f1bbc3fa"}},{"code_sha256_prefix":"15b26f6296c76fbf","entry":"cuda_kernel","repo":"MyNiuuu/MOFA-Video","repo_kind":"official","path":"MOFA-Video-Hybrid/models/softsplat.py","file_url":"https://github.com/MyNiuuu/MOFA-Video/blob/HEAD/MOFA-Video-Hybrid/models/softsplat.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15b26f6296c76fbf"}},{"code_sha256_prefix":"cf70d08839a52e41","entry":"get_sparse_flow","repo":"MyNiuuu/MOFA-Video","repo_kind":"official","path":"MOFA-Video-Hybrid/run_gradio_audio_driven.py","file_url":"https://github.com/MyNiuuu/MOFA-Video/blob/HEAD/MOFA-Video-Hybrid/run_gradio_audio_driven.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cf70d08839a52e41"}},{"code_sha256_prefix":"da94debb8019ad46","entry":"zero_module","repo":"MyNiuuu/MOFA-Video","repo_kind":"official","path":"MOFA-Video-Hybrid/models/controlnet_sdv.py","file_url":"https://github.com/MyNiuuu/MOFA-Video/blob/HEAD/MOFA-Video-Hybrid/models/controlnet_sdv.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da94debb8019ad46"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}