{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lipvoicer-generating-speech-from-silent","title":"LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading","arxiv_id":"2306.03258","date":"2023-06-05","proceeding":null,"authors":["Yochai Yemini","Aviv Shamsian","Lior Bracha","Sharon Gannot","Ethan Fetaya"],"abstract":"Lip-to-speech involves generating a natural-sounding speech synchronized with a soundless video of a person talking. Despite recent advances, current methods still cannot produce high-quality speech with high levels of intelligibility for challenging and realistic datasets such as LRS3. In this work, we present LipVoicer, a novel method that generates high-quality speech, even for in-the-wild and rich datasets, by incorporating the text modality. Given a silent video, we first predict the spoken text using a pre-trained lip-reading network. We then condition a diffusion model on the video and use the extracted text through a classifier-guidance mechanism where a pre-trained ASR serves as the classifier. LipVoicer outperforms multiple lip-to-speech baselines on LRS2 and LRS3, which are in-the-wild datasets with hundreds of unique speakers in their test set and an unrestricted vocabulary. Moreover, our experiments show that the inclusion of the text modality plays a major role in the intelligibility of the produced speech, readily perceptible while listening, and is empirically reflected in the substantial reduction of the WER metric. We demonstrate the effectiveness of LipVoicer through human evaluation, which shows that it produces more natural and synchronized speech signals compared to competing methods. Finally, we created a demo showcasing LipVoicer's superiority in producing natural, synchronized, and intelligible speech, providing additional evidence of its effectiveness. Project page and code: https://github.com/yochaiye/LipVoicer","url_abs":"https://arxiv.org/abs/2306.03258v2","url_pdf":"https://arxiv.org/pdf/2306.03258v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lipvoicer-generating-speech-from-silent","repo_url":"https://github.com/yochaiye/LipVoicer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"lip-reading","task_name":"Lip Reading"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"test","method_name":"Test"},{"method_slug":"classifier-guidance","method_name":"classifier-guidance"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.03258","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.03258"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yochaiye/LipVoicer","reach":null}],"summary":{"ran":4,"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f032578320789982","entry":"Conv","repo":"yochaiye/LipVoicer","repo_kind":"official","path":"models/wavenet.py","file_url":"https://github.com/yochaiye/LipVoicer/blob/HEAD/models/wavenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f032578320789982"}},{"code_sha256_prefix":"d728998acc977b7e","entry":"Residual_block","repo":"yochaiye/LipVoicer","repo_kind":"official","path":"models/wavenet.py","file_url":"https://github.com/yochaiye/LipVoicer/blob/HEAD/models/wavenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d728998acc977b7e"}},{"code_sha256_prefix":"7cd0b35ab0c8d4d0","entry":"Residual_group","repo":"yochaiye/LipVoicer","repo_kind":"official","path":"models/wavenet.py","file_url":"https://github.com/yochaiye/LipVoicer/blob/HEAD/models/wavenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7cd0b35ab0c8d4d0"}},{"code_sha256_prefix":"66e65ec45aa60f3f","entry":"ZeroConv1d","repo":"yochaiye/LipVoicer","repo_kind":"official","path":"models/wavenet.py","file_url":"https://github.com/yochaiye/LipVoicer/blob/HEAD/models/wavenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66e65ec45aa60f3f"}},{"code_sha256_prefix":"71f7821aa75eba0e","entry":"calc_diffusion_step_embedding","repo":"yochaiye/LipVoicer","repo_kind":"official","path":"models/wavenet.py","file_url":"https://github.com/yochaiye/LipVoicer/blob/HEAD/models/wavenet.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71f7821aa75eba0e"}},{"code_sha256_prefix":"c0206614f3d486ea","entry":"WaveNet","repo":"yochaiye/LipVoicer","repo_kind":"official","path":"models/wavenet.py","file_url":"https://github.com/yochaiye/LipVoicer/blob/HEAD/models/wavenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0206614f3d486ea"}},{"code_sha256_prefix":"4f55e2fd270242aa","entry":"sampling","repo":"yochaiye/LipVoicer","repo_kind":"official","path":"inference_real_video.py","file_url":"https://github.com/yochaiye/LipVoicer/blob/HEAD/inference_real_video.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f55e2fd270242aa"}},{"code_sha256_prefix":"d0162a52c3fb7098","entry":"training_loss","repo":"yochaiye/LipVoicer","repo_kind":"official","path":"train_melgen.py","file_url":"https://github.com/yochaiye/LipVoicer/blob/HEAD/train_melgen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0162a52c3fb7098"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}