{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/meshtalk-3d-face-animation-from-speech-using","title":"MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement","arxiv_id":"2104.08223","date":"2021-04-16","proceeding":"ICCV 2021 10","authors":["Alexander Richard","Michael Zollhoefer","Yandong Wen","Fernando de la Torre","Yaser Sheikh"],"abstract":"This paper presents a generic method for generating full facial 3D animation from speech. Existing approaches to audio-driven facial animation exhibit uncanny or static upper face animation, fail to produce accurate and plausible co-articulation or rely on person-specific models that limit their scalability. To improve upon existing models, we propose a generic audio-driven facial animation approach that achieves highly realistic motion synthesis results for the entire face. At the core of our approach is a categorical latent space for facial animation that disentangles audio-correlated and audio-uncorrelated information based on a novel cross-modality loss. Our approach ensures highly accurate lip motion, while also synthesizing plausible animation of the parts of the face that are uncorrelated to the audio signal, such as eye blinks and eye brow motion. We demonstrate that our approach outperforms several baselines and obtains state-of-the-art quality both qualitatively and quantitatively. A perceptual user study demonstrates that our approach is deemed more realistic than the current state-of-the-art in over 75% of cases. We recommend watching the supplemental video before reading the paper: https://github.com/facebookresearch/meshtalk","url_abs":"https://arxiv.org/abs/2104.08223v2","url_pdf":"https://arxiv.org/pdf/2104.08223v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"meshtalk-3d-face-animation-from-speech-using","repo_url":"https://github.com/facebookresearch/meshtalk","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"meshtalk-3d-face-animation-from-speech-using","repo_url":"https://github.com/facebookresearch/multiface","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"3d-face-animation","task_name":"3D Face Animation"},{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"motion-synthesis","task_name":"Motion Synthesis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-face-animation-on-vocaset","task":"3D Face Animation","dataset":"VOCASET","model":"MeshTalk","rank_in_archive_order":2,"of":2,"metrics":{"Lip Vertex Error":"6.7436"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.08223","atlas_url":"https://app.syntology.ai/?focus=2104.08223","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08223"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/multiface","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/meshtalk","reach":null}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"4c8c05e1023ec3bf","entry":"MaskedContextConvolution","repo":"facebookresearch/meshtalk","repo_kind":"official","path":"models/context_model.py","file_url":"https://github.com/facebookresearch/meshtalk/blob/HEAD/models/context_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4c8c05e1023ec3bf"}},{"code_sha256_prefix":"68f4a07643467bd2","entry":"Net","repo":"facebookresearch/meshtalk","repo_kind":"official","path":"models/context_model.py","file_url":"https://github.com/facebookresearch/meshtalk/blob/HEAD/models/context_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"68f4a07643467bd2"}},{"code_sha256_prefix":"1fc5329a0f56fa8a","entry":"ContextModel","repo":"facebookresearch/meshtalk","repo_kind":"official","path":"models/context_model.py","file_url":"https://github.com/facebookresearch/meshtalk/blob/HEAD/models/context_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1fc5329a0f56fa8a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}