{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/speech-resynthesis-from-discrete-disentangled","title":"Speech Resynthesis from Discrete Disentangled Self-Supervised Representations","arxiv_id":"2104.00355","date":"2021-04-01","proceeding":null,"authors":["Adam Polyak","Yossi Adi","Jade Copet","Eugene Kharitonov","Kushal Lakhotia","Wei-Ning Hsu","Abdelrahman Mohamed","Emmanuel Dupoux"],"abstract":"We propose using self-supervised discrete representations for the task of speech resynthesis. To generate disentangled representation, we separately extract low-bitrate representations for speech content, prosodic information, and speaker identity. This allows to synthesize speech in a controllable manner. We analyze various state-of-the-art, self-supervised representation learning methods and shed light on the advantages of each method while considering reconstruction quality and disentanglement properties. Specifically, we evaluate the F0 reconstruction, speaker identification performance (for both resynthesis and voice conversion), recordings' intelligibility, and overall quality using subjective human evaluation. Lastly, we demonstrate how these representations can be used for an ultra-lightweight speech codec. Using the obtained representations, we can get to a rate of 365 bits per second while providing better speech quality than the baseline methods. Audio samples can be found under the following link: speechbot.github.io/resynthesis.","url_abs":"https://arxiv.org/abs/2104.00355v3","url_pdf":"https://arxiv.org/pdf/2104.00355v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"speech-resynthesis-from-discrete-disentangled","repo_url":"https://github.com/facebookresearch/speech-resynthesis","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"speech-resynthesis-from-discrete-disentangled","repo_url":"https://github.com/inconnu11/Objective-evaluation_speech_synthesis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"resynthesis","task_name":"Resynthesis"},{"task_slug":"speaker-identification","task_name":"Speaker Identification"},{"task_slug":"voice-conversion","task_name":"Voice Conversion"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2104.00355","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.00355"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/inconnu11/Objective-evaluation_speech_synthesis","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/speech-resynthesis","reach":null}],"summary":{"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"e1e231f2b37db769","entry":"load_checkpoint","repo":"facebookresearch/speech-resynthesis","repo_kind":"official","path":"inference.py","file_url":"https://github.com/facebookresearch/speech-resynthesis/blob/HEAD/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e1e231f2b37db769"}},{"code_sha256_prefix":"d9005824ac9e4969","entry":"progbar","repo":"facebookresearch/speech-resynthesis","repo_kind":"official","path":"inference.py","file_url":"https://github.com/facebookresearch/speech-resynthesis/blob/HEAD/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d9005824ac9e4969"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}