{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lip-to-speech-synthesis-with-visual-context-1","title":"Lip to Speech Synthesis with Visual Context Attentional GAN","arxiv_id":"2204.01726","date":"2022-04-04","proceeding":"NeurIPS 2021 12","authors":["Minsu Kim","Joanna Hong","Yong Man Ro"],"abstract":"In this paper, we propose a novel lip-to-speech generative adversarial network, Visual Context Attentional GAN (VCA-GAN), which can jointly model local and global lip movements during speech synthesis. Specifically, the proposed VCA-GAN synthesizes the speech from local lip visual features by finding a mapping function of viseme-to-phoneme, while global visual context is embedded into the intermediate layers of the generator to clarify the ambiguity in the mapping induced by homophene. To achieve this, a visual context attention module is proposed where it encodes global representations from the local visual features, and provides the desired global visual context corresponding to the given coarse speech representation to the generator through audio-visual attention. In addition to the explicit modelling of local and global visual representations, synchronization learning is introduced as a form of contrastive learning that guides the generator to synthesize a speech in sync with the given input lip movements. Extensive experiments demonstrate that the proposed VCA-GAN outperforms existing state-of-the-art and is able to effectively synthesize the speech from multi-speaker that has been barely handled in the previous works.","url_abs":"https://arxiv.org/abs/2204.01726v1","url_pdf":"https://arxiv.org/pdf/2204.01726v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lip-to-speech-synthesis-with-visual-context-1","repo_url":"https://github.com/ms-dot-k/Visual-Context-Attentional-GAN","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":null,"task_name":"Generative Adversarial Network"},{"task_slug":"lip-to-speech-synthesis","task_name":"Lip to Speech Synthesis"},{"task_slug":"speech-synthesis","task_name":"Speech Synthesis"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.01726","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.01726"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ms-dot-k/Visual-Context-Attentional-GAN","reach":null}],"summary":{"ran":2,"ran_violates":1,"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"9d7ac4b6650a733e","entry":"AVAttention","repo":"ms-dot-k/Visual-Context-Attentional-GAN","repo_kind":"official","path":"src/models/generator.py","file_url":"https://github.com/ms-dot-k/Visual-Context-Attentional-GAN/blob/HEAD/src/models/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9d7ac4b6650a733e"}},{"code_sha256_prefix":"b12f75a3084b70bc","entry":"GenResBlk","repo":"ms-dot-k/Visual-Context-Attentional-GAN","repo_kind":"official","path":"src/models/generator.py","file_url":"https://github.com/ms-dot-k/Visual-Context-Attentional-GAN/blob/HEAD/src/models/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b12f75a3084b70bc"}},{"code_sha256_prefix":"e12ff8f4b54c91ab","entry":"final_length","repo":"ms-dot-k/Visual-Context-Attentional-GAN","repo_kind":"official","path":"src/models/generator.py","file_url":"https://github.com/ms-dot-k/Visual-Context-Attentional-GAN/blob/HEAD/src/models/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e12ff8f4b54c91ab"}},{"code_sha256_prefix":"15299c0f13e547a6","entry":"gan_loss","repo":"ms-dot-k/Visual-Context-Attentional-GAN","repo_kind":"official","path":"src/models/generator.py","file_url":"https://github.com/ms-dot-k/Visual-Context-Attentional-GAN/blob/HEAD/src/models/generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"15299c0f13e547a6"}},{"code_sha256_prefix":"0876d64985952bda","entry":"Decoder","repo":"ms-dot-k/Visual-Context-Attentional-GAN","repo_kind":"official","path":"src/models/generator.py","file_url":"https://github.com/ms-dot-k/Visual-Context-Attentional-GAN/blob/HEAD/src/models/generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0876d64985952bda"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}