{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/av2av-direct-audio-visual-speech-to-audio","title":"AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation","arxiv_id":"2312.02512","date":"2023-12-05","proceeding":"CVPR 2024 1","authors":["Jeongsoo Choi","Se Jin Park","Minsu Kim","Yong Man Ro"],"abstract":"This paper proposes a novel direct Audio-Visual Speech to Audio-Visual Speech Translation (AV2AV) framework, where the input and output of the system are multimodal (i.e., audio and visual speech). With the proposed AV2AV, two key advantages can be brought: 1) We can perform real-like conversations with individuals worldwide in a virtual meeting by utilizing our own primary languages. In contrast to Speech-to-Speech Translation (A2A), which solely translates between audio modalities, the proposed AV2AV directly translates between audio-visual speech. This capability enhances the dialogue experience by presenting synchronized lip movements along with the translated speech. 2) We can improve the robustness of the spoken language translation system. By employing the complementary information of audio-visual speech, the system can effectively translate spoken language even in the presence of acoustic noise, showcasing robust performance. To mitigate the problem of the absence of a parallel AV2AV translation dataset, we propose to train our spoken language translation system with the audio-only dataset of A2A. This is done by learning unified audio-visual speech representations through self-supervised learning in advance to train the translation system. Moreover, we propose an AV-Renderer that can generate raw audio and video in parallel. It is designed with zero-shot speaker modeling, thus the speaker in source audio-visual speech can be maintained at the target translated audio-visual speech. The effectiveness of AV2AV is evaluated with extensive experiments in a many-to-many language translation setting. Demo page is available on https://choijeongsoo.github.io/av2av.","url_abs":"https://arxiv.org/abs/2312.02512v2","url_pdf":"https://arxiv.org/pdf/2312.02512v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"av2av-direct-audio-visual-speech-to-audio","repo_url":"https://github.com/choijeongsoo/av2av","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"speech-to-speech-translation","task_name":"Speech-to-Speech Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.02512","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.02512"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/choijeongsoo/av2av","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"583f9780bdd00a45","entry":"conv3x3","repo":"choijeongsoo/av2av","repo_kind":"official","path":"av2unit/avhubert/resnet.py","file_url":"https://github.com/choijeongsoo/av2av/blob/HEAD/av2unit/avhubert/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"583f9780bdd00a45"}},{"code_sha256_prefix":"d50e3a8f0b0f9a21","entry":"downsample_basic_block","repo":"choijeongsoo/av2av","repo_kind":"official","path":"av2unit/avhubert/resnet.py","file_url":"https://github.com/choijeongsoo/av2av/blob/HEAD/av2unit/avhubert/resnet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d50e3a8f0b0f9a21"}},{"code_sha256_prefix":"546a770034e200d1","entry":"downsample_basic_block_v2","repo":"choijeongsoo/av2av","repo_kind":"official","path":"av2unit/avhubert/resnet.py","file_url":"https://github.com/choijeongsoo/av2av/blob/HEAD/av2unit/avhubert/resnet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"546a770034e200d1"}},{"code_sha256_prefix":"bb62e9d4f15b339c","entry":"process_units","repo":"choijeongsoo/av2av","repo_kind":"official","path":"util.py","file_url":"https://github.com/choijeongsoo/av2av/blob/HEAD/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb62e9d4f15b339c"}},{"code_sha256_prefix":"f0641584d0fe0fb5","entry":"load_audio_visual","repo":"choijeongsoo/av2av","repo_kind":"official","path":"av2unit/avhubert/hubert_dataset.py","file_url":"https://github.com/choijeongsoo/av2av/blob/HEAD/av2unit/avhubert/hubert_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0641584d0fe0fb5"}},{"code_sha256_prefix":"113a523619225ba0","entry":"load_label","repo":"choijeongsoo/av2av","repo_kind":"official","path":"av2unit/avhubert/hubert_dataset.py","file_url":"https://github.com/choijeongsoo/av2av/blob/HEAD/av2unit/avhubert/hubert_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"113a523619225ba0"}},{"code_sha256_prefix":"3e9450b6ac2fcddd","entry":"load_label_offset","repo":"choijeongsoo/av2av","repo_kind":"official","path":"av2unit/avhubert/hubert_dataset.py","file_url":"https://github.com/choijeongsoo/av2av/blob/HEAD/av2unit/avhubert/hubert_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e9450b6ac2fcddd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}