{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/megactor-harness-the-power-of-raw-video-for","title":"MegActor: Harness the Power of Raw Video for Vivid Portrait Animation","arxiv_id":"2405.20851","date":"2024-05-31","proceeding":null,"authors":["Shurong Yang","Huadong Li","Juhao Wu","Minhao Jing","Linze Li","Renhe Ji","Jiajun Liang","Haoqiang Fan"],"abstract":"Despite raw driving videos contain richer information on facial expressions than intermediate representations such as landmarks in the field of portrait animation, they are seldom the subject of research. This is due to two challenges inherent in portrait animation driven with raw videos: 1) significant identity leakage; 2) Irrelevant background and facial details such as wrinkles degrade performance. To harnesses the power of the raw videos for vivid portrait animation, we proposed a pioneering conditional diffusion model named as MegActor. First, we introduced a synthetic data generation framework for creating videos with consistent motion and expressions but inconsistent IDs to mitigate the issue of ID leakage. Second, we segmented the foreground and background of the reference image and employed CLIP to encode the background details. This encoded information is then integrated into the network via a text embedding module, thereby ensuring the stability of the background. Finally, we further style transfer the appearance of the reference image to the driving video to eliminate the influence of facial details in the driving videos. Our final model was trained solely on public datasets, achieving results comparable to commercial models. We hope this will help the open-source community.The code is available at https://github.com/megvii-research/MegFaceAnimate.","url_abs":"https://arxiv.org/abs/2405.20851v2","url_pdf":"https://arxiv.org/pdf/2405.20851v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"megactor-harness-the-power-of-raw-video-for","repo_url":"https://github.com/megvii-research/megactor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"megactor-harness-the-power-of-raw-video-for","repo_url":"https://github.com/megvii-research/megfaceanimate","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"portrait-animation","task_name":"Portrait Animation"},{"task_slug":"style-transfer","task_name":"Style Transfer"},{"task_slug":"synthetic-data-generation","task_name":"Synthetic Data Generation"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.20851","atlas_url":"https://app.syntology.ai/?focus=2405.20851","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.20851"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/megvii-research/megactor","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/megvii-research/megfaceanimate","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran":7,"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"official":{"samples":17,"ran":9,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f9ec7d70add02b6f","entry":"HWC3","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"controlnet_aux_lib/util.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/controlnet_aux_lib/util.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9ec7d70add02b6f"}},{"code_sha256_prefix":"45f1cb4de58b3f3d","entry":"get_2d_sincos_pos_embed","repo":"megvii-research/megactor","repo_kind":"official","path":"animate/megactor-sigma/embeddings.py","file_url":"https://github.com/megvii-research/megactor/blob/HEAD/animate/megactor-sigma/embeddings.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"45f1cb4de58b3f3d"}},{"code_sha256_prefix":"d0e5d8aa76eec8a1","entry":"get_context_scheduler","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"animate/context.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/animate/context.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d0e5d8aa76eec8a1"}},{"code_sha256_prefix":"859e29c1dc1b42e9","entry":"get_total_steps","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"animate/context.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/animate/context.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"859e29c1dc1b42e9"}},{"code_sha256_prefix":"cb55313c2e3ca6a8","entry":"make_noise_disk","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"controlnet_aux_lib/util.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/controlnet_aux_lib/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb55313c2e3ca6a8"}},{"code_sha256_prefix":"ef55c657c8ef7120","entry":"ordered_halving","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"animate/context.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/animate/context.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef55c657c8ef7120"}},{"code_sha256_prefix":"ee0d00edd4592e12","entry":"simulate_head_motion","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"eval_audio.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/eval_audio.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ee0d00edd4592e12"}},{"code_sha256_prefix":"8b497c21236ea29d","entry":"train_collate_fn","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"face_dataset.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/face_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b497c21236ea29d"}},{"code_sha256_prefix":"4719c763c53be3fe","entry":"zero_module","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"animate/motion_module.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/animate/motion_module.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4719c763c53be3fe"}},{"code_sha256_prefix":"59fa7b38de05f443","entry":"get_2d_sincos_pos_embed_from_grid","repo":"megvii-research/megactor","repo_kind":"official","path":"animate/megactor-sigma/embeddings.py","file_url":"https://github.com/megvii-research/megactor/blob/HEAD/animate/megactor-sigma/embeddings.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"59fa7b38de05f443"}},{"code_sha256_prefix":"1db2c40c2d2bd67b","entry":"get_clip_frames","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"_preprocess.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/_preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1db2c40c2d2bd67b"}},{"code_sha256_prefix":"f350ab376ca5e41c","entry":"get_packed_frames","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"_preprocess.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/_preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f350ab376ca5e41c"}},{"code_sha256_prefix":"1dd5f2925621c29e","entry":"get_packed_frames","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"_preprocess.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/_preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1dd5f2925621c29e"}},{"code_sha256_prefix":"38f69a3d1e46016d","entry":"get_timestep_embedding","repo":"megvii-research/megactor","repo_kind":"official","path":"animate/megactor-sigma/embeddings.py","file_url":"https://github.com/megvii-research/megactor/blob/HEAD/animate/megactor-sigma/embeddings.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38f69a3d1e46016d"}},{"code_sha256_prefix":"6cce57d1077b8c49","entry":"nms","repo":"megvii-research/megfaceanimate","repo_kind":"official","path":"controlnet_aux_lib/util.py","file_url":"https://github.com/megvii-research/megfaceanimate/blob/HEAD/controlnet_aux_lib/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6cce57d1077b8c49"}},{"code_sha256_prefix":"e55f24470648724c","entry":"torch_dfs","repo":"megvii-research/megactor","repo_kind":"official","path":"animate/megactor-sigma/stable_diffusion_controlnet_reference.py","file_url":"https://github.com/megvii-research/megactor/blob/HEAD/animate/megactor-sigma/stable_diffusion_controlnet_reference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e55f24470648724c"}},{"code_sha256_prefix":"da94debb8019ad46","entry":"zero_module","repo":"megvii-research/megactor","repo_kind":"official","path":"animate/megactor-sigma/controlnet.py","file_url":"https://github.com/megvii-research/megactor/blob/HEAD/animate/megactor-sigma/controlnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da94debb8019ad46"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}