{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/speech2lip-high-fidelity-speech-to-lip","title":"Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video","arxiv_id":"2309.04814","date":"2023-09-09","proceeding":"ICCV 2023 1","authors":["Xiuzhe Wu","Pengfei Hu","Yang Wu","Xiaoyang Lyu","Yan-Pei Cao","Ying Shan","Wenming Yang","Zhongqian Sun","Xiaojuan Qi"],"abstract":"Synthesizing realistic videos according to a given speech is still an open challenge. Previous works have been plagued by issues such as inaccurate lip shape generation and poor image quality. The key reason is that only motions and appearances on limited facial areas (e.g., lip area) are mainly driven by the input speech. Therefore, directly learning a mapping function from speech to the entire head image is prone to ambiguity, particularly when using a short video for training. We thus propose a decomposition-synthesis-composition framework named Speech to Lip (Speech2Lip) that disentangles speech-sensitive and speech-insensitive motion/appearance to facilitate effective learning from limited training data, resulting in the generation of natural-looking videos. First, given a fixed head pose (i.e., canonical space), we present a speech-driven implicit model for lip image generation which concentrates on learning speech-sensitive motion and appearance. Next, to model the major speech-insensitive motion (i.e., head movement), we introduce a geometry-aware mutual explicit mapping (GAMEM) module that establishes geometric mappings between different head poses. This allows us to paste generated lip images at the canonical space onto head images with arbitrary poses and synthesize talking videos with natural head movements. In addition, a Blend-Net and a contrastive sync loss are introduced to enhance the overall synthesis performance. Quantitative and qualitative results on three benchmarks demonstrate that our model can be trained by a video of just a few minutes in length and achieve state-of-the-art performance in both visual quality and speech-visual synchronization. Code: https://github.com/CVMI-Lab/Speech2Lip.","url_abs":"https://arxiv.org/abs/2309.04814v1","url_pdf":"https://arxiv.org/pdf/2309.04814v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"speech2lip-high-fidelity-speech-to-lip","repo_url":"https://github.com/cvmi-lab/speech2lip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2309.04814","atlas_url":"https://app.syntology.ai/?focus=2309.04814","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.04814"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/CVMI-Lab/Speech2Lip","reach":null}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"db678981cfa67937","entry":"Embedder","repo":"CVMI-Lab/Speech2Lip","repo_kind":"official","path":"src/face_simple/models/tf_nerf.py","file_url":"https://github.com/CVMI-Lab/Speech2Lip/blob/HEAD/src/face_simple/models/tf_nerf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db678981cfa67937"}},{"code_sha256_prefix":"4826917b383dd206","entry":"PositionalEncodingTime","repo":"CVMI-Lab/Speech2Lip","repo_kind":"official","path":"src/face_simple/models/tf_nerf.py","file_url":"https://github.com/CVMI-Lab/Speech2Lip/blob/HEAD/src/face_simple/models/tf_nerf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4826917b383dd206"}},{"code_sha256_prefix":"6d17597e4d506981","entry":"SimpleUnetLight","repo":"CVMI-Lab/Speech2Lip","repo_kind":"official","path":"src/face_simple/models/tf_nerf.py","file_url":"https://github.com/CVMI-Lab/Speech2Lip/blob/HEAD/src/face_simple/models/tf_nerf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d17597e4d506981"}},{"code_sha256_prefix":"133e0e9522dc539c","entry":"Up","repo":"CVMI-Lab/Speech2Lip","repo_kind":"official","path":"src/face_simple/models/tf_nerf.py","file_url":"https://github.com/CVMI-Lab/Speech2Lip/blob/HEAD/src/face_simple/models/tf_nerf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"133e0e9522dc539c"}},{"code_sha256_prefix":"3362299eda81959f","entry":"TalkingFace","repo":"CVMI-Lab/Speech2Lip","repo_kind":"official","path":"src/face_simple/models/tf_nerf.py","file_url":"https://github.com/CVMI-Lab/Speech2Lip/blob/HEAD/src/face_simple/models/tf_nerf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3362299eda81959f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}