{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-analysis-and-synthesis-reconstructing","title":"Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations","arxiv_id":"2110.14513","date":"2021-10-27","proceeding":"NeurIPS 2021 12","authors":["Hyeong-Seok Choi","Juheon Lee","Wansoo Kim","Jie Hwan Lee","Hoon Heo","Kyogu Lee"],"abstract":"We present a neural analysis and synthesis (NANSY) framework that can manipulate voice, pitch, and speed of an arbitrary speech signal. Most of the previous works have focused on using information bottleneck to disentangle analysis features for controllable synthesis, which usually results in poor reconstruction quality. We address this issue by proposing a novel training strategy based on information perturbation. The idea is to perturb information in the original input signal (e.g., formant, pitch, and frequency response), thereby letting synthesis networks selectively take essential attributes to reconstruct the input signal. Because NANSY does not need any bottleneck structures, it enjoys both high reconstruction quality and controllability. Furthermore, NANSY does not require any labels associated with speech data such as text and speaker information, but rather uses a new set of analysis features, i.e., wav2vec feature and newly proposed pitch feature, Yingram, which allows for fully self-supervised training. Taking advantage of fully self-supervised training, NANSY can be easily extended to a multilingual setting by simply training it with a multilingual dataset. The experiments show that NANSY can achieve significant improvement in performance in several applications such as zero-shot voice conversion, pitch shift, and time-scale modification.","url_abs":"https://arxiv.org/abs/2110.14513v2","url_pdf":"https://arxiv.org/pdf/2110.14513v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-analysis-and-synthesis-reconstructing","repo_url":"https://github.com/dhchoi99/NANSY","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"neural-analysis-and-synthesis-reconstructing","repo_url":"https://github.com/revsic/torch-nansy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"voice-conversion","task_name":"Voice Conversion"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2110.14513","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.14513"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dhchoi99/NANSY","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/revsic/torch-nansy","reach":null}],"summary":{"ran":8,"unverified":2},"by_repo_kind":{"listed":{"samples":10,"ran":8,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"f9343c6bc90f8cd4","entry":"ConditionalLayerNorm","repo":"revsic/torch-nansy","repo_kind":"listed","path":"nansy/synthesizer.py","file_url":"https://github.com/revsic/torch-nansy/blob/HEAD/nansy/synthesizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9343c6bc90f8cd4"}},{"code_sha256_prefix":"544e239a538d061c","entry":"ConditionalLayerNorm","repo":"dhchoi99/NANSY","repo_kind":"listed","path":"models/synthesis.py","file_url":"https://github.com/dhchoi99/NANSY/blob/HEAD/models/synthesis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"544e239a538d061c"}},{"code_sha256_prefix":"04c8057fc6f2deff","entry":"ConvGLU","repo":"revsic/torch-nansy","repo_kind":"listed","path":"nansy/synthesizer.py","file_url":"https://github.com/revsic/torch-nansy/blob/HEAD/nansy/synthesizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04c8057fc6f2deff"}},{"code_sha256_prefix":"af41e4f52f661656","entry":"ConvGLU","repo":"dhchoi99/NANSY","repo_kind":"listed","path":"models/synthesis.py","file_url":"https://github.com/dhchoi99/NANSY/blob/HEAD/models/synthesis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af41e4f52f661656"}},{"code_sha256_prefix":"e9e0b2afcfc28781","entry":"PreConv","repo":"dhchoi99/NANSY","repo_kind":"listed","path":"models/synthesis.py","file_url":"https://github.com/dhchoi99/NANSY/blob/HEAD/models/synthesis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9e0b2afcfc28781"}},{"code_sha256_prefix":"a2ce4f0df0596d33","entry":"ResBlock1","repo":"dhchoi99/NANSY","repo_kind":"listed","path":"models/synthesis.py","file_url":"https://github.com/dhchoi99/NANSY/blob/HEAD/models/synthesis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2ce4f0df0596d33"}},{"code_sha256_prefix":"2330fc6e8150cf5e","entry":"ResBlock2","repo":"dhchoi99/NANSY","repo_kind":"listed","path":"models/synthesis.py","file_url":"https://github.com/dhchoi99/NANSY/blob/HEAD/models/synthesis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2330fc6e8150cf5e"}},{"code_sha256_prefix":"d2d2661a86323397","entry":"Synthesizer","repo":"revsic/torch-nansy","repo_kind":"listed","path":"nansy/synthesizer.py","file_url":"https://github.com/revsic/torch-nansy/blob/HEAD/nansy/synthesizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2d2661a86323397"}},{"code_sha256_prefix":"aedea83432badf8d","entry":"Generator","repo":"dhchoi99/NANSY","repo_kind":"listed","path":"models/synthesis.py","file_url":"https://github.com/dhchoi99/NANSY/blob/HEAD/models/synthesis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aedea83432badf8d"}},{"code_sha256_prefix":"2c98d43d678e4da1","entry":"Synthesis","repo":"dhchoi99/NANSY","repo_kind":"listed","path":"models/synthesis.py","file_url":"https://github.com/dhchoi99/NANSY/blob/HEAD/models/synthesis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2c98d43d678e4da1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}